[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980527431848-22676-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:35:27 dist-test-slave-5lgp krb5kdc[22683](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:35:27 dist-test-slave-5lgp krb5kdc[22683](info): set up 2 sockets
Jun 20 18:35:27 dist-test-slave-5lgp krb5kdc[22683](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260620 18:35:28.565699 22676 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.006s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980527431848-22676-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980527431848-22676-0/krb5kdc/test-user.keytab.
Jun 20 18:35:28 dist-test-slave-5lgp krb5kdc[22683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:35:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:35:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.37.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.37.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.37.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab.
Entry for principal kudu/127.22.37.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab.
I20260620 18:35:28.732879 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980527431848-22676-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980527431848-22676-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980527431848-22676-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.62:41101
--webserver_interface=127.22.37.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab
--principal=kudu/127.22.37.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:41105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.37.62:41101
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:35:29.057709 22696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:35:29.058003 22696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:35:29.058171 22696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:35:29.065527 22696 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:35:29.065650 22696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:35:29.065768 22696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:35:29.065851 22696 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:35:29.065932 22696 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:35:29.075964 22696 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:41105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980527431848-22676-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980527431848-22676-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.37.62:41101
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab
--principal=kudu/127.22.37.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.62:41101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.37.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980527431848-22676-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:35:29.077517 22696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:35:29.078886 22696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:35:29.091642 22701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:29.091667 22702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:29.091584 22704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:35:29.091951 22696 server_base.cc:1034] running on GCE node
I20260620 18:35:29.092917 22696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:35:29.094380 22696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:35:29.095583 22696 hybrid_clock.cc:648] HybridClock initialized: now 1781980529095514 us; error 58 us; skew 500 ppm
I20260620 18:35:29.095878 22696 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:35:29 dist-test-slave-5lgp krb5kdc[22683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980529, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:35:29.098963 22696 init.cc:377] Logged in from keytab as kudu/127.22.37.62@KRBTEST.COM (short username kudu)
I20260620 18:35:29.100469 22696 webserver.cc:460] Webserver started at http://127.22.37.62:41403/ using document root <none> and password file <none>
I20260620 18:35:29.101135 22696 fs_manager.cc:359] Metadata directory not provided
I20260620 18:35:29.101233 22696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:35:29.101557 22696 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:35:29.104449 22696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980527431848-22676-0/minicluster-data/master-0/data/instance:
uuid: "7b272c2bab1748b4aa54c1f141c3739f"
format_stamp: "Formatted at 2026-06-20 18:35:29 on dist-test-slave-5lgp"
server_key: "d50e780d0ee841adf9dda83e15b7ac4a"
server_key_iv: "850dc58a5f03267913e3095f7aa38290"
server_key_version: "encryptionkey@0"
I20260620 18:35:29.105167 22696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980527431848-22676-0/minicluster-data/master-0/wal/instance:
uuid: "7b272c2bab1748b4aa54c1f141c3739f"
format_stamp: "Formatted at 2026-06-20 18:35:29 on dist-test-slave-5lgp"
server_key: "d50e780d0ee841adf9dda83e15b7ac4a"
server_key_iv: "850dc58a5f03267913e3095f7aa38290"
server_key_version: "encryptionkey@0"
I20260620 18:35:29.110608 22696 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 18:35:29.114326 22711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:29.115728 22696 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:35:29.115871 22696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980527431848-22676-0/minicluster-data/master-0/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980527431848-22676-0/minicluster-data/master-0/wal
uuid: "7b272c2bab1748b4aa54c1f141c3739f"
format_stamp: "Formatted at 2026-06-20 18:35:29 on dist-test-slave-5lgp"
server_key: "d50e780d0ee841adf9dda83e15b7ac4a"
server_key_iv: "850dc58a5f03267913e3095f7aa38290"
server_key_version: "encryptionkey@0"
I20260620 18:35:29.116048 22696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980527431848-22676-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980527431848-22676-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980527431848-22676-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:35:29.134634 22696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:35:29.139816 22696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:35:29.140175 22696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:35:29.160701 22696 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.62:41101
I20260620 18:35:29.160737 22762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.62:41101 every 8 connection(s)
I20260620 18:35:29.162357 22696 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
I20260620 18:35:29.166350 22763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:35:29.169961 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 22696
I20260620 18:35:29.170290 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980527431848-22676-0/minicluster-data/master-0/wal/instance
I20260620 18:35:29.172734 22676 external_mini_cluster.cc:1428] Setting key ff24522724c26b87d3f782143f9d8660
I20260620 18:35:29.179286 22763 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b272c2bab1748b4aa54c1f141c3739f: Bootstrap starting.
I20260620 18:35:29.182483 22763 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b272c2bab1748b4aa54c1f141c3739f: Neither blocks nor log segments found. Creating new log.
I20260620 18:35:29.183534 22763 log.cc:824] T 00000000000000000000000000000000 P 7b272c2bab1748b4aa54c1f141c3739f: Log is configured to *not* fsync() on all Append() calls
I20260620 18:35:29.186455 22763 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b272c2bab1748b4aa54c1f141c3739f: No bootstrap required, opened a new log
I20260620 18:35:29.196702 22763 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b272c2bab1748b4aa54c1f141c3739f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b272c2bab1748b4aa54c1f141c3739f" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 41101 } }
I20260620 18:35:29.197233 22763 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7b272c2bab1748b4aa54c1f141c3739f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:35:29.197346 22763 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b272c2bab1748b4aa54c1f141c3739f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b272c2bab1748b4aa54c1f141c3739f, State: Initialized, Role: FOLLOWER
I20260620 18:35:29.198150 22763 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b272c2bab1748b4aa54c1f141c3739f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b272c2bab1748b4aa54c1f141c3739f" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 41101 } }
I20260620 18:35:29.198449 22763 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7b272c2bab1748b4aa54c1f141c3739f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:35:29.198616 22763 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b272c2bab1748b4aa54c1f141c3739f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:35:29.198813 22763 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b272c2bab1748b4aa54c1f141c3739f [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:35:29.201236 22763 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b272c2bab1748b4aa54c1f141c3739f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b272c2bab1748b4aa54c1f141c3739f" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 41101 } }
I20260620 18:35:29.201802 22763 leader_election.cc:302] T 00000000000000000000000000000000 P 7b272c2bab1748b4aa54c1f141c3739f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b272c2bab1748b4aa54c1f141c3739f; no voters:
I20260620 18:35:29.202392 22763 leader_election.cc:288] T 00000000000000000000000000000000 P 7b272c2bab1748b4aa54c1f141c3739f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:35:29.202544 22768 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b272c2bab1748b4aa54c1f141c3739f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:35:29.203193 22768 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b272c2bab1748b4aa54c1f141c3739f [term 1 LEADER]: Becoming Leader. State: Replica: 7b272c2bab1748b4aa54c1f141c3739f, State: Running, Role: LEADER
I20260620 18:35:29.203845 22768 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b272c2bab1748b4aa54c1f141c3739f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b272c2bab1748b4aa54c1f141c3739f" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 41101 } }
I20260620 18:35:29.204663 22763 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b272c2bab1748b4aa54c1f141c3739f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:35:29.206703 22770 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b272c2bab1748b4aa54c1f141c3739f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b272c2bab1748b4aa54c1f141c3739f. Latest consensus state: current_term: 1 leader_uuid: "7b272c2bab1748b4aa54c1f141c3739f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b272c2bab1748b4aa54c1f141c3739f" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 41101 } } }
I20260620 18:35:29.207089 22770 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b272c2bab1748b4aa54c1f141c3739f [sys.catalog]: This master's current role is: LEADER
I20260620 18:35:29.207651 22769 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b272c2bab1748b4aa54c1f141c3739f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7b272c2bab1748b4aa54c1f141c3739f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b272c2bab1748b4aa54c1f141c3739f" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 41101 } } }
Jun 20 18:35:29 dist-test-slave-5lgp krb5kdc[22683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM
I20260620 18:35:29.207919 22769 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b272c2bab1748b4aa54c1f141c3739f [sys.catalog]: This master's current role is: LEADER
I20260620 18:35:29.219965 22773 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:35:29.224802 22765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:29.177613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34194 (local address 127.22.37.62:41101)
0620 18:35:29.178579 (+ 966us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:29.178595 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:29.179973 (+ 1378us) server_negotiation.cc:407] Connection header received
0620 18:35:29.189112 (+ 9139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:29.189142 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:29.189720 (+ 578us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:29.190165 (+ 445us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:29.193035 (+ 2870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:29.195224 (+ 2189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:29.196962 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:29.198056 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:29.198257 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:29.209976 (+ 11719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:29.209992 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:29.210019 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:29.210094 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:29.215285 (+ 5191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:29.216735 (+ 1450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:29.216749 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:29.216760 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:29.217131 (+ 371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:29.217877 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:29.217886 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:29.217892 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:29.218950 (+ 1058us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:29.222358 (+ 3408us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:29.222686 (+ 328us) server_negotiation.cc:299] Negotiation successful
0620 18:35:29.223216 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":715,"thread_start_us":471,"threads_started":1}
I20260620 18:35:29.226999 22773 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:35:29.239043 22773 catalog_manager.cc:1347] Generated new cluster ID: abe930d282a1485e873ab1c06fae6b55
I20260620 18:35:29.239187 22773 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:35:29.256870 22773 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:35:29.258302 22773 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:35:29.268522 22773 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7b272c2bab1748b4aa54c1f141c3739f: Generated new TSK 0
I20260620 18:35:29.269578 22773 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.37.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.37.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.37.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.1.keytab.
Entry for principal kudu/127.22.37.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.1.keytab.
I20260620 18:35:29.322405 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980527431848-22676-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980527431848-22676-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980527431848-22676-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980527431848-22676-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.37.1:0
--local_ip_for_outbound_sockets=127.22.37.1
--webserver_interface=127.22.37.1
--webserver_port=0
--tserver_master_addrs=127.22.37.62:41101
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.1.keytab
--principal=kudu/127.22.37.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:41105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:35:29.665361 22790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:35:29.665782 22790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:35:29.665973 22790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:35:29.673342 22790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:35:29.673573 22790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:35:29.673777 22790 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.37.1
I20260620 18:35:29.685316 22790 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:41105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980527431848-22676-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980527431848-22676-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.1.keytab
--principal=kudu/127.22.37.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.37.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980527431848-22676-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.37.1
--webserver_port=0
--tserver_master_addrs=127.22.37.62:41101
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.37.1
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980527431848-22676-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:35:29.686751 22790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:35:29.688201 22790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:35:29.701992 22795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:29.702028 22796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:29.702090 22798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:35:29.703017 22790 server_base.cc:1034] running on GCE node
I20260620 18:35:29.703660 22790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:35:29.704630 22790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:35:29.705938 22790 hybrid_clock.cc:648] HybridClock initialized: now 1781980529705840 us; error 92 us; skew 500 ppm
I20260620 18:35:29.706267 22790 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:35:29 dist-test-slave-5lgp krb5kdc[22683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980529, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:35:29.709250 22790 init.cc:377] Logged in from keytab as kudu/127.22.37.1@KRBTEST.COM (short username kudu)
I20260620 18:35:29.710721 22790 webserver.cc:460] Webserver started at http://127.22.37.1:38523/ using document root <none> and password file <none>
I20260620 18:35:29.711388 22790 fs_manager.cc:359] Metadata directory not provided
I20260620 18:35:29.711526 22790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:35:29.711831 22790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:35:29.714958 22790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980527431848-22676-0/minicluster-data/ts-0/data/instance:
uuid: "3a0c6b63d2d54714a5032ce268b4d81c"
format_stamp: "Formatted at 2026-06-20 18:35:29 on dist-test-slave-5lgp"
server_key: "6dd65336db6141051440269b56f781ed"
server_key_iv: "63a08a103d34048083f1ca42d66f9a14"
server_key_version: "encryptionkey@0"
I20260620 18:35:29.715700 22790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980527431848-22676-0/minicluster-data/ts-0/wal/instance:
uuid: "3a0c6b63d2d54714a5032ce268b4d81c"
format_stamp: "Formatted at 2026-06-20 18:35:29 on dist-test-slave-5lgp"
server_key: "6dd65336db6141051440269b56f781ed"
server_key_iv: "63a08a103d34048083f1ca42d66f9a14"
server_key_version: "encryptionkey@0"
I20260620 18:35:29.720798 22790 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260620 18:35:29.724176 22805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:29.725523 22790 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:35:29.725679 22790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980527431848-22676-0/minicluster-data/ts-0/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980527431848-22676-0/minicluster-data/ts-0/wal
uuid: "3a0c6b63d2d54714a5032ce268b4d81c"
format_stamp: "Formatted at 2026-06-20 18:35:29 on dist-test-slave-5lgp"
server_key: "6dd65336db6141051440269b56f781ed"
server_key_iv: "63a08a103d34048083f1ca42d66f9a14"
server_key_version: "encryptionkey@0"
I20260620 18:35:29.725863 22790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980527431848-22676-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980527431848-22676-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980527431848-22676-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:35:29.761170 22790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:35:29.765666 22790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:35:29.765959 22790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:35:29.766942 22790 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:35:29.768537 22790 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:35:29.768638 22790 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:29.768790 22790 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:35:29.768860 22790 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:29.802811 22790 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.1:36625
I20260620 18:35:29.802886 22917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.1:36625 every 8 connection(s)
I20260620 18:35:29.804148 22790 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980527431848-22676-0/minicluster-data/ts-0/data/info.pb
I20260620 18:35:29.812335 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 22790
I20260620 18:35:29.812567 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980527431848-22676-0/minicluster-data/ts-0/wal/instance
I20260620 18:35:29.813314 22676 external_mini_cluster.cc:1428] Setting key 47fc791cf14b6b2f3e6a0cb17cddabc7
Jun 20 18:35:29 dist-test-slave-5lgp krb5kdc[22683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980529, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.1@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM
I20260620 18:35:29.825810 22920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:29.806880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.1:38871 (local address 127.22.37.62:41101)
0620 18:35:29.807324 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:29.807328 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:29.808200 (+ 872us) server_negotiation.cc:407] Connection header received
0620 18:35:29.810075 (+ 1875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:29.810080 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:29.810170 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:29.810298 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:29.811820 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:29.812945 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:29.815042 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:29.815819 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:29.815969 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:29.819610 (+ 3641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:29.819617 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:29.819626 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:29.819670 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:29.822457 (+ 2787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:29.823375 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:29.823379 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:29.823381 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:29.823507 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:29.823991 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:29.823994 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:29.823996 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:29.824241 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:29.824348 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:29.825295 (+ 947us) server_negotiation.cc:299] Negotiation successful
0620 18:35:29.825533 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":231,"threads_started":1}
I20260620 18:35:29.827438 22921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:29.807329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:41101 (local address 127.22.37.1:38871)
0620 18:35:29.808016 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:29.808060 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:29.809809 (+ 1749us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:29.810560 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:29.810568 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:29.811084 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:29.811638 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:29.811656 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:29.813126 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:29.813131 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:29.814871 (+ 1740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:29.814885 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:29.816060 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:29.816065 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:29.816277 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:29.817062 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:35:29.817088 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:29.819370 (+ 2282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:35:29.822595 (+ 3225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:29.822607 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:29.822627 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:29.823253 (+ 626us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:29.823617 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:29.823625 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:29.823631 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:29.823881 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:29.824361 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:35:29.824372 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:35:29.824869 (+ 497us) client_negotiation.cc:769] Sending connection context
0620 18:35:29.825200 (+ 331us) client_negotiation.cc:240] Negotiation successful
0620 18:35:29.825837 (+ 637us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":288,"threads_started":1}
I20260620 18:35:29.829195 22918 heartbeater.cc:347] Connected to a master server at 127.22.37.62:41101
I20260620 18:35:29.829599 22918 heartbeater.cc:464] Registering TS with master...
I20260620 18:35:29.830448 22918 heartbeater.cc:511] Master 127.22.37.62:41101 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.22.37.2@KRBTEST.COM; defaulting to no policy
I20260620 18:35:29.832860 22728 ts_manager.cc:194] Registered new tserver with Master: 3a0c6b63d2d54714a5032ce268b4d81c (127.22.37.1:36625)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.37.2@KRBTEST.COM" created.
I20260620 18:35:29.834496 22728 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.37.1@KRBTEST.COM'} at 127.22.37.1:38871
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.37.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.2.keytab.
Entry for principal kudu/127.22.37.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.2.keytab.
I20260620 18:35:29.856288 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980527431848-22676-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980527431848-22676-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980527431848-22676-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980527431848-22676-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.37.2:0
--local_ip_for_outbound_sockets=127.22.37.2
--webserver_interface=127.22.37.2
--webserver_port=0
--tserver_master_addrs=127.22.37.62:41101
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.2.keytab
--principal=kudu/127.22.37.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:41105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:35:30.184665 22924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:35:30.184993 22924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:35:30.185173 22924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:35:30.192657 22924 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:35:30.192819 22924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:35:30.193009 22924 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.37.2
I20260620 18:35:30.203083 22924 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:41105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980527431848-22676-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980527431848-22676-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.2.keytab
--principal=kudu/127.22.37.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.37.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980527431848-22676-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.37.2
--webserver_port=0
--tserver_master_addrs=127.22.37.62:41101
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.37.2
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980527431848-22676-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:35:30.204454 22924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:35:30.205977 22924 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:35:30.219372 22930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:30.219327 22929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:30.219414 22932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:35:30.219967 22924 server_base.cc:1034] running on GCE node
I20260620 18:35:30.220582 22924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:35:30.221589 22924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:35:30.222826 22924 hybrid_clock.cc:648] HybridClock initialized: now 1781980530222767 us; error 57 us; skew 500 ppm
I20260620 18:35:30.223181 22924 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:35:30 dist-test-slave-5lgp krb5kdc[22683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980530, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:35:30.226684 22924 init.cc:377] Logged in from keytab as kudu/127.22.37.2@KRBTEST.COM (short username kudu)
I20260620 18:35:30.228164 22924 webserver.cc:460] Webserver started at http://127.22.37.2:46065/ using document root <none> and password file <none>
I20260620 18:35:30.228880 22924 fs_manager.cc:359] Metadata directory not provided
I20260620 18:35:30.228979 22924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:35:30.229250 22924 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:35:30.232414 22924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980527431848-22676-0/minicluster-data/ts-1/data/instance:
uuid: "23d7b5b94bf94b0a8a7018f268f0a292"
format_stamp: "Formatted at 2026-06-20 18:35:30 on dist-test-slave-5lgp"
server_key: "b2c5d22b2c4c7c0c1aaa9b78f5fc2759"
server_key_iv: "472831a3065b2c30358209696eafd355"
server_key_version: "encryptionkey@0"
I20260620 18:35:30.233273 22924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980527431848-22676-0/minicluster-data/ts-1/wal/instance:
uuid: "23d7b5b94bf94b0a8a7018f268f0a292"
format_stamp: "Formatted at 2026-06-20 18:35:30 on dist-test-slave-5lgp"
server_key: "b2c5d22b2c4c7c0c1aaa9b78f5fc2759"
server_key_iv: "472831a3065b2c30358209696eafd355"
server_key_version: "encryptionkey@0"
I20260620 18:35:30.238888 22924 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260620 18:35:30.242673 22939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:30.244195 22924 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 18:35:30.244330 22924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980527431848-22676-0/minicluster-data/ts-1/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980527431848-22676-0/minicluster-data/ts-1/wal
uuid: "23d7b5b94bf94b0a8a7018f268f0a292"
format_stamp: "Formatted at 2026-06-20 18:35:30 on dist-test-slave-5lgp"
server_key: "b2c5d22b2c4c7c0c1aaa9b78f5fc2759"
server_key_iv: "472831a3065b2c30358209696eafd355"
server_key_version: "encryptionkey@0"
I20260620 18:35:30.244582 22924 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980527431848-22676-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980527431848-22676-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980527431848-22676-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:35:30.271088 22924 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:35:30.275563 22924 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:35:30.275872 22924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:35:30.276866 22924 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:35:30.278312 22924 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:35:30.278386 22924 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:30.278527 22924 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:35:30.278587 22924 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:30.312071 22924 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.2:39961
I20260620 18:35:30.312119 23051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.2:39961 every 8 connection(s)
I20260620 18:35:30.313493 22924 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980527431848-22676-0/minicluster-data/ts-1/data/info.pb
I20260620 18:35:30.323757 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 22924
I20260620 18:35:30.323972 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980527431848-22676-0/minicluster-data/ts-1/wal/instance
I20260620 18:35:30.324410 22676 external_mini_cluster.cc:1428] Setting key 98eff801066656263080b152dfd60d73
Jun 20 18:35:30 dist-test-slave-5lgp krb5kdc[22683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980530, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.2@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM
I20260620 18:35:30.335456 22920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:30.316100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.2:32785 (local address 127.22.37.62:41101)
0620 18:35:30.316301 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:30.316305 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:30.317424 (+ 1119us) server_negotiation.cc:407] Connection header received
0620 18:35:30.319213 (+ 1789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:30.319219 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:30.319332 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:30.319447 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:30.320969 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:30.321750 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:30.323469 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:30.323963 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:30.324095 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:30.327643 (+ 3548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:30.327648 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:30.327651 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:30.327681 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:30.332239 (+ 4558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:30.333008 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:30.333016 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:30.333020 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:30.333161 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:30.333653 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:30.333660 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:30.333666 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:30.333942 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:30.334092 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:30.334825 (+ 733us) server_negotiation.cc:299] Negotiation successful
0620 18:35:30.335081 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 18:35:30.336524 23054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:30.316493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:41101 (local address 127.22.37.2:32785)
0620 18:35:30.317233 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:30.317281 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:30.318943 (+ 1662us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:30.319721 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:30.319734 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:30.320262 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:30.320803 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:30.320820 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:30.322056 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:30.322064 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:30.323309 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:30.323325 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:30.324176 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:30.324183 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:30.324409 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:30.325142 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:35:30.325171 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:30.327403 (+ 2232us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:35:30.332387 (+ 4984us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:30.332396 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:30.332422 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:30.332867 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:30.333283 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:30.333290 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:30.333294 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:30.333542 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:30.334077 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:35:30.334086 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:35:30.334462 (+ 376us) client_negotiation.cc:769] Sending connection context
0620 18:35:30.334742 (+ 280us) client_negotiation.cc:240] Negotiation successful
0620 18:35:30.335309 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":280,"threads_started":1}
I20260620 18:35:30.338253 23052 heartbeater.cc:347] Connected to a master server at 127.22.37.62:41101
I20260620 18:35:30.338614 23052 heartbeater.cc:464] Registering TS with master...
I20260620 18:35:30.339576 23052 heartbeater.cc:511] Master 127.22.37.62:41101 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.22.37.3@KRBTEST.COM; defaulting to no policy
I20260620 18:35:30.341326 22728 ts_manager.cc:194] Registered new tserver with Master: 23d7b5b94bf94b0a8a7018f268f0a292 (127.22.37.2:39961)
I20260620 18:35:30.342238 22728 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.37.2@KRBTEST.COM'} at 127.22.37.2:32785
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.37.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.37.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.3.keytab.
Entry for principal kudu/127.22.37.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.3.keytab.
I20260620 18:35:30.364795 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980527431848-22676-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980527431848-22676-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980527431848-22676-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980527431848-22676-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.37.3:0
--local_ip_for_outbound_sockets=127.22.37.3
--webserver_interface=127.22.37.3
--webserver_port=0
--tserver_master_addrs=127.22.37.62:41101
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.3.keytab
--principal=kudu/127.22.37.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:41105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:35:30.685118 23057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:35:30.685492 23057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:35:30.685668 23057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:35:30.693691 23057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:35:30.693787 23057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:35:30.694010 23057 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.37.3
I20260620 18:35:30.703922 23057 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:41105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980527431848-22676-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980527431848-22676-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.3.keytab
--principal=kudu/127.22.37.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.37.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980527431848-22676-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.37.3
--webserver_port=0
--tserver_master_addrs=127.22.37.62:41101
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.37.3
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980527431848-22676-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:35:30.705284 23057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:35:30.706709 23057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:35:30.719728 23062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:30.719689 23063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:30.719656 23065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:35:30.720933 23057 server_base.cc:1034] running on GCE node
I20260620 18:35:30.721581 23057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:35:30.722525 23057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:35:30.723766 23057 hybrid_clock.cc:648] HybridClock initialized: now 1781980530723719 us; error 56 us; skew 500 ppm
I20260620 18:35:30.724089 23057 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:35:30 dist-test-slave-5lgp krb5kdc[22683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980530, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:35:30.727090 23057 init.cc:377] Logged in from keytab as kudu/127.22.37.3@KRBTEST.COM (short username kudu)
I20260620 18:35:30.728509 23057 webserver.cc:460] Webserver started at http://127.22.37.3:36287/ using document root <none> and password file <none>
I20260620 18:35:30.729209 23057 fs_manager.cc:359] Metadata directory not provided
I20260620 18:35:30.729354 23057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:35:30.729753 23057 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:35:30.732645 23057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980527431848-22676-0/minicluster-data/ts-2/data/instance:
uuid: "9961475ed43a4421b0e8679a77c4ea8b"
format_stamp: "Formatted at 2026-06-20 18:35:30 on dist-test-slave-5lgp"
server_key: "b3d3f9c4f6ad2f622f79c0e5cd33f84d"
server_key_iv: "c4d8a9515c50bffc21414d11060efc95"
server_key_version: "encryptionkey@0"
I20260620 18:35:30.733511 23057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980527431848-22676-0/minicluster-data/ts-2/wal/instance:
uuid: "9961475ed43a4421b0e8679a77c4ea8b"
format_stamp: "Formatted at 2026-06-20 18:35:30 on dist-test-slave-5lgp"
server_key: "b3d3f9c4f6ad2f622f79c0e5cd33f84d"
server_key_iv: "c4d8a9515c50bffc21414d11060efc95"
server_key_version: "encryptionkey@0"
I20260620 18:35:30.738780 23057 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260620 18:35:30.742298 23072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:30.743584 23057 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:35:30.743759 23057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980527431848-22676-0/minicluster-data/ts-2/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980527431848-22676-0/minicluster-data/ts-2/wal
uuid: "9961475ed43a4421b0e8679a77c4ea8b"
format_stamp: "Formatted at 2026-06-20 18:35:30 on dist-test-slave-5lgp"
server_key: "b3d3f9c4f6ad2f622f79c0e5cd33f84d"
server_key_iv: "c4d8a9515c50bffc21414d11060efc95"
server_key_version: "encryptionkey@0"
I20260620 18:35:30.743957 23057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980527431848-22676-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980527431848-22676-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980527431848-22676-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:35:30.766685 23057 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:35:30.771096 23057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:35:30.771438 23057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:35:30.772445 23057 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:35:30.773859 23057 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:35:30.773965 23057 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:30.774116 23057 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:35:30.774188 23057 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:30.806705 23057 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.3:44759
I20260620 18:35:30.806794 23184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.3:44759 every 8 connection(s)
I20260620 18:35:30.807991 23057 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980527431848-22676-0/minicluster-data/ts-2/data/info.pb
I20260620 18:35:30.811192 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 23057
I20260620 18:35:30.811414 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980527431848-22676-0/minicluster-data/ts-2/wal/instance
I20260620 18:35:30.812005 22676 external_mini_cluster.cc:1428] Setting key 99f9d3eedc8705480553eacfe719d267
Jun 20 18:35:30 dist-test-slave-5lgp krb5kdc[22683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980530, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.3@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM
I20260620 18:35:30.828079 22920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:30.810560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.3:48429 (local address 127.22.37.62:41101)
0620 18:35:30.810745 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:30.810749 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:30.812090 (+ 1341us) server_negotiation.cc:407] Connection header received
0620 18:35:30.814055 (+ 1965us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:30.814061 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:30.814145 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:30.814274 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:30.816107 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:30.816857 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:30.818461 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:30.818872 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:30.819018 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:30.822533 (+ 3515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:30.822538 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:30.822541 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:30.822571 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:30.824912 (+ 2341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:30.825688 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:30.825693 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:30.825695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:30.825813 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:30.826242 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:30.826246 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:30.826249 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:30.826523 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:30.826692 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:30.827475 (+ 783us) server_negotiation.cc:299] Negotiation successful
0620 18:35:30.827692 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 18:35:30.829187 23187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:30.811123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:41101 (local address 127.22.37.3:48429)
0620 18:35:30.811929 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:30.811971 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:30.813785 (+ 1814us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:30.814506 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:30.814515 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:30.815288 (+ 773us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:30.815950 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:30.815967 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:30.817012 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:30.817022 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:30.818279 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:30.818290 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:30.819100 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:30.819108 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:30.819342 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:30.820018 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:35:30.820039 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:30.822282 (+ 2243us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:35:30.825042 (+ 2760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:30.825051 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:30.825068 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:30.825560 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:30.825939 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:30.825944 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:30.825948 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:30.826124 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:30.826658 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:35:30.826668 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:35:30.827057 (+ 389us) client_negotiation.cc:769] Sending connection context
0620 18:35:30.827370 (+ 313us) client_negotiation.cc:240] Negotiation successful
0620 18:35:30.827913 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":371,"threads_started":1}
I20260620 18:35:30.830709 23185 heartbeater.cc:347] Connected to a master server at 127.22.37.62:41101
I20260620 18:35:30.831043 23185 heartbeater.cc:464] Registering TS with master...
I20260620 18:35:30.831763 23185 heartbeater.cc:511] Master 127.22.37.62:41101 requested a full tablet report, sending...
I20260620 18:35:30.833336 22728 ts_manager.cc:194] Registered new tserver with Master: 9961475ed43a4421b0e8679a77c4ea8b (127.22.37.3:44759)
I20260620 18:35:30.834213 22728 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.37.3@KRBTEST.COM'} at 127.22.37.3:48429
I20260620 18:35:30.837503 22918 heartbeater.cc:503] Master 127.22.37.62:41101 was elected leader, sending a full tablet report...
I20260620 18:35:30.840830 22676 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 18:35:30 dist-test-slave-5lgp krb5kdc[22683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980530, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 18:35:30 dist-test-slave-5lgp krb5kdc[22683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980530, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.22.37.1@KRBTEST.COM
I20260620 18:35:30.887208 23192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:30.871023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41596 (local address 127.22.37.1:36625)
0620 18:35:30.871738 (+ 715us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:30.871748 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:30.871781 (+ 33us) server_negotiation.cc:407] Connection header received
0620 18:35:30.871869 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:30.871884 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:30.872128 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:30.872305 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:30.873856 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:30.874874 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:30.876181 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:30.876828 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:30.877111 (+ 283us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:30.881318 (+ 4207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:30.881332 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:30.881349 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:30.881435 (+ 86us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:30.884205 (+ 2770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:30.884970 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:30.884978 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:30.884981 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:30.885107 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:30.885591 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:30.885597 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:30.885600 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:30.885970 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:30.886297 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:30.886579 (+ 282us) server_negotiation.cc:299] Negotiation successful
0620 18:35:30.886850 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":346,"threads_started":1}
W20260620 18:35:30.888653 22833 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41596
Jun 20 18:35:30 dist-test-slave-5lgp krb5kdc[22683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980530, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 18:35:30 dist-test-slave-5lgp krb5kdc[22683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980530, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.37.1@KRBTEST.COM
I20260620 18:35:30.932168 23192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:30.918115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41610 (local address 127.22.37.1:36625)
0620 18:35:30.918346 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:30.918354 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:30.918677 (+ 323us) server_negotiation.cc:407] Connection header received
0620 18:35:30.918928 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:30.918932 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:30.919021 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:30.919132 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:30.920553 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:30.921261 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:30.922510 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:30.922921 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:30.923060 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:30.926902 (+ 3842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:30.926908 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:30.926911 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:30.926946 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:30.929628 (+ 2682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:30.930265 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:30.930272 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:30.930280 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:30.930442 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:30.930803 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:30.930808 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:30.930812 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:30.931072 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:30.931313 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:30.931604 (+ 291us) server_negotiation.cc:299] Negotiation successful
0620 18:35:30.931831 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 18:35:30.933781 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 22790
I20260620 18:35:30.947168 23192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:30.934869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41616 (local address 127.22.37.1:36625)
0620 18:35:30.935082 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:30.935088 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:30.935294 (+ 206us) server_negotiation.cc:407] Connection header received
0620 18:35:30.935512 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:30.935516 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:30.935572 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:30.935684 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:30.936841 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:30.937668 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:30.938954 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:30.939362 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:30.939508 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:30.941990 (+ 2482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:30.941995 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:30.941998 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:30.942032 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:30.944511 (+ 2479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:30.945177 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:30.945184 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:30.945191 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:30.945331 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:30.945751 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:30.945757 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:30.945761 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:30.946022 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:30.946268 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:30.946586 (+ 318us) server_negotiation.cc:299] Negotiation successful
0620 18:35:30.946794 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 18:35:30.947945 22913 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:35:31.032929 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 22790
I20260620 18:35:31.049830 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 22924
Jun 20 18:35:31 dist-test-slave-5lgp krb5kdc[22683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980530, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.37.2@KRBTEST.COM
I20260620 18:35:31.066763 23199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:31.050782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56118 (local address 127.22.37.2:39961)
0620 18:35:31.051344 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:31.051351 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:31.051377 (+ 26us) server_negotiation.cc:407] Connection header received
0620 18:35:31.051436 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:31.051442 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:31.051670 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:31.051807 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:31.053022 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:31.054175 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:31.055531 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:31.056132 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:31.056249 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:31.060715 (+ 4466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:31.060727 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:31.060743 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:31.060780 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:31.063752 (+ 2972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:31.064578 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:31.064585 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:31.064588 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:31.064730 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:31.065149 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:31.065152 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:31.065154 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:31.065658 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:31.065913 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:31.066213 (+ 300us) server_negotiation.cc:299] Negotiation successful
0620 18:35:31.066426 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":265,"threads_started":1}
I20260620 18:35:31.068121 23047 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:35:31.155009 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 22924
I20260620 18:35:31.172051 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 23057
Jun 20 18:35:31 dist-test-slave-5lgp krb5kdc[22683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980530, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.37.3@KRBTEST.COM
I20260620 18:35:31.188813 23201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:31.173001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34458 (local address 127.22.37.3:44759)
0620 18:35:31.173619 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:31.173626 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:31.173654 (+ 28us) server_negotiation.cc:407] Connection header received
0620 18:35:31.173714 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:31.173721 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:31.173943 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:31.174135 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:31.175487 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:31.176552 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:31.178076 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:31.178666 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:31.178819 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:31.183182 (+ 4363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:31.183195 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:31.183214 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:31.183257 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:31.186036 (+ 2779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:31.186725 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:31.186733 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:31.186736 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:31.186855 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:31.187225 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:31.187231 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:31.187234 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:31.187591 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:31.187907 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:31.188190 (+ 283us) server_negotiation.cc:299] Negotiation successful
0620 18:35:31.188461 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":274,"threads_started":1}
I20260620 18:35:31.190232 23180 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:35:31.280036 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 23057
I20260620 18:35:31.295799 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 22696
I20260620 18:35:31.297288 22758 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:35:31.402915 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 22696
2026-06-20T18:35:31Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3938 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980527431848-22676-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:35:31 dist-test-slave-5lgp krb5kdc[23206](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:35:31 dist-test-slave-5lgp krb5kdc[23206](info): set up 2 sockets
Jun 20 18:35:31 dist-test-slave-5lgp krb5kdc[23206](info): commencing operation
krb5kdc: starting...
W20260620 18:35:32.493304 22676 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980527431848-22676-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980527431848-22676-0/krb5kdc/test-user.keytab.
Jun 20 18:35:32 dist-test-slave-5lgp krb5kdc[23206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:35:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:35:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.37.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.37.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.37.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab.
Entry for principal kudu/127.22.37.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab.
I20260620 18:35:32.663358 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980527431848-22676-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980527431848-22676-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980527431848-22676-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.62:40775
--webserver_interface=127.22.37.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab
--principal=kudu/127.22.37.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:35219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.37.62:40775
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:35:32.977532 23219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:35:32.977909 23219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:35:32.978071 23219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:35:32.985178 23219 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:35:32.985332 23219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:35:32.985473 23219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:35:32.985574 23219 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:35:32.985669 23219 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:35:32.995451 23219 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:35219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980527431848-22676-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980527431848-22676-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.37.62:40775
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab
--principal=kudu/127.22.37.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.62:40775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.37.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980527431848-22676-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:35:32.996832 23219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:35:32.998219 23219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:35:33.010043 23224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:33.009997 23227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:33.009969 23225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:35:33.010803 23219 server_base.cc:1034] running on GCE node
I20260620 18:35:33.011628 23219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:35:33.013034 23219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:35:33.014259 23219 hybrid_clock.cc:648] HybridClock initialized: now 1781980533014213 us; error 57 us; skew 500 ppm
I20260620 18:35:33.014600 23219 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:35:33 dist-test-slave-5lgp krb5kdc[23206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980533, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:35:33.017721 23219 init.cc:377] Logged in from keytab as kudu/127.22.37.62@KRBTEST.COM (short username kudu)
I20260620 18:35:33.019238 23219 webserver.cc:460] Webserver started at http://127.22.37.62:43179/ using document root <none> and password file <none>
I20260620 18:35:33.020025 23219 fs_manager.cc:359] Metadata directory not provided
I20260620 18:35:33.020171 23219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:35:33.020500 23219 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:35:33.023437 23219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980527431848-22676-0/minicluster-data/master-0/data/instance:
uuid: "74c91312f2a048b4919fef8c6a4e8c1d"
format_stamp: "Formatted at 2026-06-20 18:35:33 on dist-test-slave-5lgp"
server_key: "01d78f94c4948cdd17be43e3edc09aa8"
server_key_iv: "54e9e180c547444f1b5d79372973a342"
server_key_version: "encryptionkey@0"
I20260620 18:35:33.024165 23219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980527431848-22676-0/minicluster-data/master-0/wal/instance:
uuid: "74c91312f2a048b4919fef8c6a4e8c1d"
format_stamp: "Formatted at 2026-06-20 18:35:33 on dist-test-slave-5lgp"
server_key: "01d78f94c4948cdd17be43e3edc09aa8"
server_key_iv: "54e9e180c547444f1b5d79372973a342"
server_key_version: "encryptionkey@0"
I20260620 18:35:33.029129 23219 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 18:35:33.032460 23234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:33.033785 23219 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:35:33.033962 23219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980527431848-22676-0/minicluster-data/master-0/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980527431848-22676-0/minicluster-data/master-0/wal
uuid: "74c91312f2a048b4919fef8c6a4e8c1d"
format_stamp: "Formatted at 2026-06-20 18:35:33 on dist-test-slave-5lgp"
server_key: "01d78f94c4948cdd17be43e3edc09aa8"
server_key_iv: "54e9e180c547444f1b5d79372973a342"
server_key_version: "encryptionkey@0"
I20260620 18:35:33.034158 23219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980527431848-22676-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980527431848-22676-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980527431848-22676-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:35:33.049207 23219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:35:33.053609 23219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:35:33.053911 23219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:35:33.072341 23219 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.62:40775
I20260620 18:35:33.072366 23285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.62:40775 every 8 connection(s)
I20260620 18:35:33.073807 23219 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
I20260620 18:35:33.077545 23286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:35:33.079906 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 23219
I20260620 18:35:33.080137 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980527431848-22676-0/minicluster-data/master-0/wal/instance
I20260620 18:35:33.080932 22676 external_mini_cluster.cc:1428] Setting key 2bfda5beeebea6f73d9469c9c7eab082
I20260620 18:35:33.089100 23286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74c91312f2a048b4919fef8c6a4e8c1d: Bootstrap starting.
I20260620 18:35:33.092123 23286 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 74c91312f2a048b4919fef8c6a4e8c1d: Neither blocks nor log segments found. Creating new log.
I20260620 18:35:33.093079 23286 log.cc:824] T 00000000000000000000000000000000 P 74c91312f2a048b4919fef8c6a4e8c1d: Log is configured to *not* fsync() on all Append() calls
Jun 20 18:35:33 dist-test-slave-5lgp krb5kdc[23206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM
I20260620 18:35:33.096125 23286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74c91312f2a048b4919fef8c6a4e8c1d: No bootstrap required, opened a new log
I20260620 18:35:33.103029 23289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:33.083074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40808 (local address 127.22.37.62:40775)
0620 18:35:33.083812 (+ 738us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:33.083825 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:33.083863 (+ 38us) server_negotiation.cc:407] Connection header received
0620 18:35:33.085545 (+ 1682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:33.085568 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:33.085963 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:33.086350 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:33.087800 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:33.089068 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:33.090362 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:33.091040 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:33.091230 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:33.095358 (+ 4128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:33.095368 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:33.095387 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:33.095428 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:33.098812 (+ 3384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:33.099549 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:33.099557 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:33.099563 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:33.099724 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:33.100114 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:33.100119 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:33.100124 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:33.100683 (+ 559us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:33.101004 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:33.101569 (+ 565us) server_negotiation.cc:299] Negotiation successful
0620 18:35:33.101983 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":320,"threads_started":1}
I20260620 18:35:33.103840 23286 raft_consensus.cc:348] T 00000000000000000000000000000000 P 74c91312f2a048b4919fef8c6a4e8c1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74c91312f2a048b4919fef8c6a4e8c1d" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 40775 } }
I20260620 18:35:33.104251 23286 raft_consensus.cc:374] T 00000000000000000000000000000000 P 74c91312f2a048b4919fef8c6a4e8c1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:35:33.104368 23286 raft_consensus.cc:729] T 00000000000000000000000000000000 P 74c91312f2a048b4919fef8c6a4e8c1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74c91312f2a048b4919fef8c6a4e8c1d, State: Initialized, Role: FOLLOWER
I20260620 18:35:33.104929 23286 consensus_queue.cc:260] T 00000000000000000000000000000000 P 74c91312f2a048b4919fef8c6a4e8c1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74c91312f2a048b4919fef8c6a4e8c1d" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 40775 } }
I20260620 18:35:33.105182 23286 raft_consensus.cc:388] T 00000000000000000000000000000000 P 74c91312f2a048b4919fef8c6a4e8c1d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:35:33.105324 23286 raft_consensus.cc:482] T 00000000000000000000000000000000 P 74c91312f2a048b4919fef8c6a4e8c1d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:35:33.105532 23286 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 74c91312f2a048b4919fef8c6a4e8c1d [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:35:33.107848 23286 raft_consensus.cc:504] T 00000000000000000000000000000000 P 74c91312f2a048b4919fef8c6a4e8c1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74c91312f2a048b4919fef8c6a4e8c1d" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 40775 } }
I20260620 18:35:33.108363 23286 leader_election.cc:302] T 00000000000000000000000000000000 P 74c91312f2a048b4919fef8c6a4e8c1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 74c91312f2a048b4919fef8c6a4e8c1d; no voters:
I20260620 18:35:33.108981 23286 leader_election.cc:288] T 00000000000000000000000000000000 P 74c91312f2a048b4919fef8c6a4e8c1d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:35:33.109126 23291 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 74c91312f2a048b4919fef8c6a4e8c1d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:35:33.110275 23291 raft_consensus.cc:686] T 00000000000000000000000000000000 P 74c91312f2a048b4919fef8c6a4e8c1d [term 1 LEADER]: Becoming Leader. State: Replica: 74c91312f2a048b4919fef8c6a4e8c1d, State: Running, Role: LEADER
I20260620 18:35:33.110837 23291 consensus_queue.cc:237] T 00000000000000000000000000000000 P 74c91312f2a048b4919fef8c6a4e8c1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74c91312f2a048b4919fef8c6a4e8c1d" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 40775 } }
I20260620 18:35:33.111217 23286 sys_catalog.cc:564] T 00000000000000000000000000000000 P 74c91312f2a048b4919fef8c6a4e8c1d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:35:33.113747 23292 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74c91312f2a048b4919fef8c6a4e8c1d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "74c91312f2a048b4919fef8c6a4e8c1d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74c91312f2a048b4919fef8c6a4e8c1d" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 40775 } } }
I20260620 18:35:33.114131 23292 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74c91312f2a048b4919fef8c6a4e8c1d [sys.catalog]: This master's current role is: LEADER
I20260620 18:35:33.114679 23293 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74c91312f2a048b4919fef8c6a4e8c1d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 74c91312f2a048b4919fef8c6a4e8c1d. Latest consensus state: current_term: 1 leader_uuid: "74c91312f2a048b4919fef8c6a4e8c1d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74c91312f2a048b4919fef8c6a4e8c1d" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 40775 } } }
I20260620 18:35:33.115029 23293 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74c91312f2a048b4919fef8c6a4e8c1d [sys.catalog]: This master's current role is: LEADER
I20260620 18:35:33.115705 23300 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:35:33.121397 23300 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:35:33.130789 23300 catalog_manager.cc:1347] Generated new cluster ID: bd89f46f87cd460fa3d6ad76610ec242
I20260620 18:35:33.130926 23300 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:35:33.144306 23300 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:35:33.146171 23300 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:35:33.158066 23300 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 74c91312f2a048b4919fef8c6a4e8c1d: Generated new TSK 0
I20260620 18:35:33.158964 23300 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.37.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.37.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.37.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.1.keytab.
Entry for principal kudu/127.22.37.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.1.keytab.
I20260620 18:35:33.202059 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980527431848-22676-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980527431848-22676-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980527431848-22676-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980527431848-22676-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.37.1:0
--local_ip_for_outbound_sockets=127.22.37.1
--webserver_interface=127.22.37.1
--webserver_port=0
--tserver_master_addrs=127.22.37.62:40775
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.1.keytab
--principal=kudu/127.22.37.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:35219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:35:33.542464 23312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:35:33.542724 23312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:35:33.542819 23312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:35:33.549893 23312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:35:33.550001 23312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:35:33.550115 23312 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.37.1
I20260620 18:35:33.560055 23312 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:35219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980527431848-22676-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980527431848-22676-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.1.keytab
--principal=kudu/127.22.37.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.37.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980527431848-22676-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.37.1
--webserver_port=0
--tserver_master_addrs=127.22.37.62:40775
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.37.1
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980527431848-22676-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:35:33.561381 23312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:35:33.562698 23312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:35:33.575630 23317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:33.575670 23318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:33.575573 23320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:35:33.576716 23312 server_base.cc:1034] running on GCE node
I20260620 18:35:33.577255 23312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:35:33.578213 23312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:35:33.579417 23312 hybrid_clock.cc:648] HybridClock initialized: now 1781980533579368 us; error 44 us; skew 500 ppm
I20260620 18:35:33.579684 23312 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:35:33 dist-test-slave-5lgp krb5kdc[23206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980533, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:35:33.583263 23312 init.cc:377] Logged in from keytab as kudu/127.22.37.1@KRBTEST.COM (short username kudu)
I20260620 18:35:33.584728 23312 webserver.cc:460] Webserver started at http://127.22.37.1:42295/ using document root <none> and password file <none>
I20260620 18:35:33.585484 23312 fs_manager.cc:359] Metadata directory not provided
I20260620 18:35:33.585624 23312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:35:33.585935 23312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:35:33.588753 23312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980527431848-22676-0/minicluster-data/ts-0/data/instance:
uuid: "b645921494d34e1ca613978ba9206b37"
format_stamp: "Formatted at 2026-06-20 18:35:33 on dist-test-slave-5lgp"
server_key: "b802bb27cb9613ceacb434a874e04ea4"
server_key_iv: "4bd3d38bf6799e8eaf23a42c41699158"
server_key_version: "encryptionkey@0"
I20260620 18:35:33.589607 23312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980527431848-22676-0/minicluster-data/ts-0/wal/instance:
uuid: "b645921494d34e1ca613978ba9206b37"
format_stamp: "Formatted at 2026-06-20 18:35:33 on dist-test-slave-5lgp"
server_key: "b802bb27cb9613ceacb434a874e04ea4"
server_key_iv: "4bd3d38bf6799e8eaf23a42c41699158"
server_key_version: "encryptionkey@0"
I20260620 18:35:33.594578 23312 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 18:35:33.597947 23327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:33.599318 23312 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:35:33.599519 23312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980527431848-22676-0/minicluster-data/ts-0/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980527431848-22676-0/minicluster-data/ts-0/wal
uuid: "b645921494d34e1ca613978ba9206b37"
format_stamp: "Formatted at 2026-06-20 18:35:33 on dist-test-slave-5lgp"
server_key: "b802bb27cb9613ceacb434a874e04ea4"
server_key_iv: "4bd3d38bf6799e8eaf23a42c41699158"
server_key_version: "encryptionkey@0"
I20260620 18:35:33.599723 23312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980527431848-22676-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980527431848-22676-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980527431848-22676-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:35:33.614444 23312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:35:33.618616 23312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:35:33.618916 23312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:35:33.619869 23312 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:35:33.621237 23312 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:35:33.621337 23312 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:33.621493 23312 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:35:33.621572 23312 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:33.654546 23312 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.1:35027
I20260620 18:35:33.654587 23439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.1:35027 every 8 connection(s)
I20260620 18:35:33.655853 23312 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980527431848-22676-0/minicluster-data/ts-0/data/info.pb
I20260620 18:35:33.661525 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 23312
I20260620 18:35:33.661741 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980527431848-22676-0/minicluster-data/ts-0/wal/instance
I20260620 18:35:33.662372 22676 external_mini_cluster.cc:1428] Setting key 9228910de1bc39e4869e1e825eca648e
Jun 20 18:35:33 dist-test-slave-5lgp krb5kdc[23206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980533, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.1@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM
I20260620 18:35:33.676537 23442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:33.658443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.1:39119 (local address 127.22.37.62:40775)
0620 18:35:33.658885 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:33.658890 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:33.659902 (+ 1012us) server_negotiation.cc:407] Connection header received
0620 18:35:33.661870 (+ 1968us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:33.661876 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:33.661972 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:33.662095 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:33.663815 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:33.664525 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:33.666165 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:33.666579 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:33.666716 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:33.670673 (+ 3957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:33.670680 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:33.670684 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:33.670726 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:33.673318 (+ 2592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:33.674174 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:33.674180 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:33.674184 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:33.674311 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:33.674748 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:33.674754 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:33.674757 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:33.675032 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:33.675193 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:33.675906 (+ 713us) server_negotiation.cc:299] Negotiation successful
0620 18:35:33.676168 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":237,"threads_started":1}
WARNING: no policy specified for kudu/127.22.37.2@KRBTEST.COM; defaulting to no policy
I20260620 18:35:33.677692 23443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:33.658937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:40775 (local address 127.22.37.1:39119)
0620 18:35:33.659663 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:33.659706 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:33.661540 (+ 1834us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:33.662341 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:33.662355 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:33.662998 (+ 643us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:33.663641 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:33.663659 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:33.664685 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:33.664691 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:33.666019 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:33.666030 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:33.666803 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:33.666811 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:33.667051 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:33.667749 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:35:33.667770 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:33.670422 (+ 2652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:35:33.673560 (+ 3138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:33.673571 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:33.673589 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:33.674039 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:33.674423 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:33.674428 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:33.674432 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:33.674628 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:33.675159 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:35:33.675169 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:35:33.675538 (+ 369us) client_negotiation.cc:769] Sending connection context
0620 18:35:33.675808 (+ 270us) client_negotiation.cc:240] Negotiation successful
0620 18:35:33.676400 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":284,"threads_started":1}
I20260620 18:35:33.679281 23440 heartbeater.cc:347] Connected to a master server at 127.22.37.62:40775
I20260620 18:35:33.679620 23440 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.37.2@KRBTEST.COM" created.
I20260620 18:35:33.680553 23440 heartbeater.cc:511] Master 127.22.37.62:40775 requested a full tablet report, sending...
I20260620 18:35:33.682848 23251 ts_manager.cc:194] Registered new tserver with Master: b645921494d34e1ca613978ba9206b37 (127.22.37.1:35027)
I20260620 18:35:33.684410 23251 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.37.1@KRBTEST.COM'} at 127.22.37.1:39119
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.37.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.2.keytab.
Entry for principal kudu/127.22.37.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.2.keytab.
I20260620 18:35:33.700950 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980527431848-22676-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980527431848-22676-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980527431848-22676-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980527431848-22676-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.37.2:0
--local_ip_for_outbound_sockets=127.22.37.2
--webserver_interface=127.22.37.2
--webserver_port=0
--tserver_master_addrs=127.22.37.62:40775
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.2.keytab
--principal=kudu/127.22.37.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:35219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:35:34.019344 23446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:35:34.019616 23446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:35:34.019776 23446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:35:34.027084 23446 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:35:34.027182 23446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:35:34.027393 23446 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.37.2
I20260620 18:35:34.036819 23446 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:35219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980527431848-22676-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980527431848-22676-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.2.keytab
--principal=kudu/127.22.37.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.37.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980527431848-22676-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.37.2
--webserver_port=0
--tserver_master_addrs=127.22.37.62:40775
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.37.2
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980527431848-22676-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:35:34.038285 23446 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:35:34.039595 23446 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:35:34.052182 23452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:34.052274 23454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:34.052199 23451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:35:34.053056 23446 server_base.cc:1034] running on GCE node
I20260620 18:35:34.053781 23446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:35:34.054687 23446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:35:34.055881 23446 hybrid_clock.cc:648] HybridClock initialized: now 1781980534055842 us; error 32 us; skew 500 ppm
I20260620 18:35:34.056205 23446 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:35:34 dist-test-slave-5lgp krb5kdc[23206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980534, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:35:34.059201 23446 init.cc:377] Logged in from keytab as kudu/127.22.37.2@KRBTEST.COM (short username kudu)
I20260620 18:35:34.060556 23446 webserver.cc:460] Webserver started at http://127.22.37.2:45099/ using document root <none> and password file <none>
I20260620 18:35:34.061224 23446 fs_manager.cc:359] Metadata directory not provided
I20260620 18:35:34.061408 23446 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:35:34.061726 23446 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:35:34.064499 23446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980527431848-22676-0/minicluster-data/ts-1/data/instance:
uuid: "0ac8f29722034c799021207e3c6a9dcc"
format_stamp: "Formatted at 2026-06-20 18:35:34 on dist-test-slave-5lgp"
server_key: "9bd51d3537470213da23c1e16fed0977"
server_key_iv: "8faeca029f3ff3e7662abfc597c09c13"
server_key_version: "encryptionkey@0"
I20260620 18:35:34.065279 23446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980527431848-22676-0/minicluster-data/ts-1/wal/instance:
uuid: "0ac8f29722034c799021207e3c6a9dcc"
format_stamp: "Formatted at 2026-06-20 18:35:34 on dist-test-slave-5lgp"
server_key: "9bd51d3537470213da23c1e16fed0977"
server_key_iv: "8faeca029f3ff3e7662abfc597c09c13"
server_key_version: "encryptionkey@0"
I20260620 18:35:34.070178 23446 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 18:35:34.073755 23461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:34.075139 23446 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 18:35:34.075300 23446 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980527431848-22676-0/minicluster-data/ts-1/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980527431848-22676-0/minicluster-data/ts-1/wal
uuid: "0ac8f29722034c799021207e3c6a9dcc"
format_stamp: "Formatted at 2026-06-20 18:35:34 on dist-test-slave-5lgp"
server_key: "9bd51d3537470213da23c1e16fed0977"
server_key_iv: "8faeca029f3ff3e7662abfc597c09c13"
server_key_version: "encryptionkey@0"
I20260620 18:35:34.075471 23446 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980527431848-22676-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980527431848-22676-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980527431848-22676-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:35:34.097199 23446 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:35:34.101084 23446 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:35:34.101421 23446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:35:34.102386 23446 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:35:34.103700 23446 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:35:34.103808 23446 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:34.103950 23446 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:35:34.104020 23446 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:34.138242 23446 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.2:38349
I20260620 18:35:34.138278 23573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.2:38349 every 8 connection(s)
I20260620 18:35:34.139565 23446 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980527431848-22676-0/minicluster-data/ts-1/data/info.pb
I20260620 18:35:34.147873 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 23446
I20260620 18:35:34.148089 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980527431848-22676-0/minicluster-data/ts-1/wal/instance
I20260620 18:35:34.148520 22676 external_mini_cluster.cc:1428] Setting key b1ff371f1d6d2839f009ebcb45c7235d
Jun 20 18:35:34 dist-test-slave-5lgp krb5kdc[23206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980534, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.2@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM
I20260620 18:35:34.160808 23442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:34.142350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.2:53111 (local address 127.22.37.62:40775)
0620 18:35:34.142577 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:34.142584 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:34.144043 (+ 1459us) server_negotiation.cc:407] Connection header received
0620 18:35:34.146072 (+ 2029us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:34.146077 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:34.146150 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:34.146243 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:34.147752 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:34.148620 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:34.150775 (+ 2155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:34.151210 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:34.151346 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:34.154895 (+ 3549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:34.154900 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:34.154904 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:34.154934 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:34.157582 (+ 2648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:34.158543 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:34.158547 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:34.158549 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:34.158675 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:34.159153 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:34.159156 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:34.159158 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:34.159396 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:34.159499 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:34.160390 (+ 891us) server_negotiation.cc:299] Negotiation successful
0620 18:35:34.160567 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260620 18:35:34.162341 23576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:34.142976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:40775 (local address 127.22.37.2:53111)
0620 18:35:34.143862 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:34.143900 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:34.145791 (+ 1891us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:34.146464 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:34.146472 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:34.146989 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:34.147579 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:34.147596 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:34.148751 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:34.148756 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:34.150632 (+ 1876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:34.150650 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:34.151435 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:34.151442 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:34.151682 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:34.152368 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:35:34.152395 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:34.154618 (+ 2223us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:35:34.157721 (+ 3103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:34.157732 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:34.157753 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:34.158409 (+ 656us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:34.158782 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:34.158789 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:34.158796 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:34.159044 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:34.159523 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:35:34.159533 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:35:34.159994 (+ 461us) client_negotiation.cc:769] Sending connection context
0620 18:35:34.160318 (+ 324us) client_negotiation.cc:240] Negotiation successful
0620 18:35:34.160906 (+ 588us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"thread_start_us":343,"threads_started":1}
I20260620 18:35:34.164088 23574 heartbeater.cc:347] Connected to a master server at 127.22.37.62:40775
I20260620 18:35:34.164395 23574 heartbeater.cc:464] Registering TS with master...
I20260620 18:35:34.165103 23574 heartbeater.cc:511] Master 127.22.37.62:40775 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.22.37.3@KRBTEST.COM; defaulting to no policy
I20260620 18:35:34.166682 23251 ts_manager.cc:194] Registered new tserver with Master: 0ac8f29722034c799021207e3c6a9dcc (127.22.37.2:38349)
I20260620 18:35:34.167454 23251 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.37.2@KRBTEST.COM'} at 127.22.37.2:53111
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.37.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.37.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.3.keytab.
Entry for principal kudu/127.22.37.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.3.keytab.
I20260620 18:35:34.190647 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980527431848-22676-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980527431848-22676-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980527431848-22676-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980527431848-22676-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.37.3:0
--local_ip_for_outbound_sockets=127.22.37.3
--webserver_interface=127.22.37.3
--webserver_port=0
--tserver_master_addrs=127.22.37.62:40775
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.3.keytab
--principal=kudu/127.22.37.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:35219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:35:34.512425 23579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:35:34.512825 23579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:35:34.513015 23579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:35:34.520442 23579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:35:34.520557 23579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:35:34.520740 23579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.37.3
I20260620 18:35:34.530843 23579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:35219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980527431848-22676-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980527431848-22676-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.3.keytab
--principal=kudu/127.22.37.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.37.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980527431848-22676-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.37.3
--webserver_port=0
--tserver_master_addrs=127.22.37.62:40775
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.37.3
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980527431848-22676-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:35:34.532209 23579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:35:34.533571 23579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:35:34.547051 23584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:34.547075 23585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:34.547425 23587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:35:34.548483 23579 server_base.cc:1034] running on GCE node
I20260620 18:35:34.549132 23579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:35:34.550189 23579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:35:34.551424 23579 hybrid_clock.cc:648] HybridClock initialized: now 1781980534551375 us; error 46 us; skew 500 ppm
I20260620 18:35:34.551754 23579 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:35:34 dist-test-slave-5lgp krb5kdc[23206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980534, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:35:34.559985 23579 init.cc:377] Logged in from keytab as kudu/127.22.37.3@KRBTEST.COM (short username kudu)
I20260620 18:35:34.562000 23579 webserver.cc:460] Webserver started at http://127.22.37.3:45117/ using document root <none> and password file <none>
I20260620 18:35:34.562737 23579 fs_manager.cc:359] Metadata directory not provided
I20260620 18:35:34.562886 23579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:35:34.563217 23579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:35:34.566340 23579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980527431848-22676-0/minicluster-data/ts-2/data/instance:
uuid: "8001e68b4b3f4b06bd13ca50dad66fe8"
format_stamp: "Formatted at 2026-06-20 18:35:34 on dist-test-slave-5lgp"
server_key: "7904f9a7773343d97047f8782c94f8a3"
server_key_iv: "a83f7ae9117af5c2248e996b4f96ad85"
server_key_version: "encryptionkey@0"
I20260620 18:35:34.567158 23579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980527431848-22676-0/minicluster-data/ts-2/wal/instance:
uuid: "8001e68b4b3f4b06bd13ca50dad66fe8"
format_stamp: "Formatted at 2026-06-20 18:35:34 on dist-test-slave-5lgp"
server_key: "7904f9a7773343d97047f8782c94f8a3"
server_key_iv: "a83f7ae9117af5c2248e996b4f96ad85"
server_key_version: "encryptionkey@0"
I20260620 18:35:34.572309 23579 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 18:35:34.575764 23594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:34.577144 23579 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:35:34.577319 23579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980527431848-22676-0/minicluster-data/ts-2/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980527431848-22676-0/minicluster-data/ts-2/wal
uuid: "8001e68b4b3f4b06bd13ca50dad66fe8"
format_stamp: "Formatted at 2026-06-20 18:35:34 on dist-test-slave-5lgp"
server_key: "7904f9a7773343d97047f8782c94f8a3"
server_key_iv: "a83f7ae9117af5c2248e996b4f96ad85"
server_key_version: "encryptionkey@0"
I20260620 18:35:34.577574 23579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980527431848-22676-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980527431848-22676-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980527431848-22676-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:35:34.598232 23579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:35:34.602444 23579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:35:34.602762 23579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:35:34.603926 23579 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:35:34.605329 23579 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:35:34.605459 23579 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260620 18:35:34.605624 23579 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:35:34.605705 23579 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:34.639384 23579 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.3:46523
I20260620 18:35:34.639441 23706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.3:46523 every 8 connection(s)
I20260620 18:35:34.640722 23579 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980527431848-22676-0/minicluster-data/ts-2/data/info.pb
I20260620 18:35:34.648068 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 23579
I20260620 18:35:34.648283 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980527431848-22676-0/minicluster-data/ts-2/wal/instance
I20260620 18:35:34.648742 22676 external_mini_cluster.cc:1428] Setting key 532ed38d5d1969f35a6dd25206bed289
Jun 20 18:35:34 dist-test-slave-5lgp krb5kdc[23206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980534, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.3@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM
I20260620 18:35:34.661883 23442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:34.643434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.3:45387 (local address 127.22.37.62:40775)
0620 18:35:34.643611 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:34.643616 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:34.644893 (+ 1277us) server_negotiation.cc:407] Connection header received
0620 18:35:34.646859 (+ 1966us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:34.646865 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:34.646982 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:34.647100 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:34.648869 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:34.649806 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:34.651596 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:34.652171 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:34.652319 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:34.656081 (+ 3762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:34.656086 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:34.656089 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:34.656123 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:34.658944 (+ 2821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:34.659769 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:34.659773 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:34.659776 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:34.659895 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:34.660376 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:34.660379 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:34.660381 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:34.660641 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:34.660758 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:34.661458 (+ 700us) server_negotiation.cc:299] Negotiation successful
0620 18:35:34.661626 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 18:35:34.663010 23709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:34.643988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:40775 (local address 127.22.37.3:45387)
0620 18:35:34.644709 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:34.644754 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:34.646532 (+ 1778us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:34.647379 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:34.647389 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:34.647956 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:34.648667 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:34.648686 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:34.649939 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:34.649951 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:34.651449 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:34.651461 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:34.652414 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:34.652422 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:34.652665 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:34.653322 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:35:34.653350 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:34.655858 (+ 2508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:35:34.659105 (+ 3247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:34.659114 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:34.659130 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:34.659664 (+ 534us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:34.660084 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:34.660088 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:34.660091 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:34.660276 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:34.660748 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:35:34.660754 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:35:34.661047 (+ 293us) client_negotiation.cc:769] Sending connection context
0620 18:35:34.661332 (+ 285us) client_negotiation.cc:240] Negotiation successful
0620 18:35:34.661788 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":271,"threads_started":1}
I20260620 18:35:34.664369 23707 heartbeater.cc:347] Connected to a master server at 127.22.37.62:40775
I20260620 18:35:34.664682 23707 heartbeater.cc:464] Registering TS with master...
I20260620 18:35:34.665445 23707 heartbeater.cc:511] Master 127.22.37.62:40775 requested a full tablet report, sending...
I20260620 18:35:34.666822 23251 ts_manager.cc:194] Registered new tserver with Master: 8001e68b4b3f4b06bd13ca50dad66fe8 (127.22.37.3:46523)
I20260620 18:35:34.667620 23251 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.37.3@KRBTEST.COM'} at 127.22.37.3:45387
I20260620 18:35:34.676975 22676 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:35:34.687353 23440 heartbeater.cc:503] Master 127.22.37.62:40775 was elected leader, sending a full tablet report...
I20260620 18:35:34.695966 23442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:34.683426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40820 (local address 127.22.37.62:40775)
0620 18:35:34.683596 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:34.683601 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:34.684098 (+ 497us) server_negotiation.cc:407] Connection header received
0620 18:35:34.684242 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:34.684246 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:34.684312 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:34.684396 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:34.685869 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:34.686610 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:34.687931 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:34.688521 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:34.688652 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:34.691178 (+ 2526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:34.691182 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:34.691186 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:34.691217 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:34.693624 (+ 2407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:34.694242 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:34.694246 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:34.694248 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:34.694376 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:34.694688 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:34.694691 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:34.694694 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:34.694989 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:34.695196 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:34.695425 (+ 229us) server_negotiation.cc:299] Negotiation successful
0620 18:35:34.695631 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 18:35:34.702921 23251 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40820:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:35:34.709735 23251 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:35:34.733428 23720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:34.721952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55960 (local address 127.22.37.3:46523)
0620 18:35:34.722898 (+ 946us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:34.722906 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:34.723345 (+ 439us) server_negotiation.cc:407] Connection header received
0620 18:35:34.723883 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:34.723895 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:34.724229 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:34.725302 (+ 1073us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:35:34.725649 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:34.727739 (+ 2090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:34.730639 (+ 2900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:34.732117 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:34.732248 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:34.732507 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:34.732898 (+ 391us) server_negotiation.cc:299] Negotiation successful
0620 18:35:34.733071 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":765,"thread_start_us":629,"threads_started":1}
I20260620 18:35:34.733531 23719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:34.721756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.3:46523 (local address 127.0.0.1:55960)
0620 18:35:34.723015 (+ 1259us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:34.723249 (+ 234us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:34.723580 (+ 331us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:34.724494 (+ 914us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:34.724504 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:34.724543 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:35:34.725051 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:34.725073 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:34.727902 (+ 2829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:34.727911 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:34.730467 (+ 2556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:34.730481 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:34.732347 (+ 1866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:34.732360 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:34.732639 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:34.732698 (+ 59us) client_negotiation.cc:769] Sending connection context
0620 18:35:34.732839 (+ 141us) client_negotiation.cc:240] Negotiation successful
0620 18:35:34.733028 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"mutex_wait_us":21,"thread_start_us":1046,"threads_started":1}
I20260620 18:35:34.736645 23723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:34.723820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.2:38349 (local address 127.0.0.1:50062)
0620 18:35:34.728662 (+ 4842us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:34.728678 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:34.728802 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:34.729660 (+ 858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:34.729664 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:34.729690 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:35:34.730107 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:34.730117 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:34.731699 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:34.731704 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:34.734014 (+ 2310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:34.734028 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:34.735841 (+ 1813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:34.735849 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:34.736004 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:34.736036 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 18:35:34.736121 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 18:35:34.736253 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4711,"mutex_wait_us":42,"thread_start_us":4576,"threads_started":1}
I20260620 18:35:34.736882 23724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:34.724241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50062 (local address 127.22.37.2:38349)
0620 18:35:34.724750 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:34.724761 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:34.728776 (+ 4015us) server_negotiation.cc:407] Connection header received
0620 18:35:34.728975 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:34.728981 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:34.729180 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:34.729389 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:35:34.730298 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:34.731577 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:34.734174 (+ 2597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:34.735636 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:34.735763 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:34.735947 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:34.736211 (+ 264us) server_negotiation.cc:299] Negotiation successful
0620 18:35:34.736378 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":268,"threads_started":1}
I20260620 18:35:34.737272 23721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:34.722624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37760 (local address 127.22.37.1:35027)
0620 18:35:34.723266 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:34.723274 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:34.728493 (+ 5219us) server_negotiation.cc:407] Connection header received
0620 18:35:34.728777 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:34.728785 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:34.729012 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:34.729214 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:35:34.729947 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:34.731471 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:34.733992 (+ 2521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:34.735636 (+ 1644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:34.735755 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:34.735970 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:34.736763 (+ 793us) server_negotiation.cc:299] Negotiation successful
0620 18:35:34.736943 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":597,"threads_started":1}
I20260620 18:35:34.737439 23722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:34.723171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.1:35027 (local address 127.0.0.1:37760)
0620 18:35:34.728368 (+ 5197us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:34.728389 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:34.728590 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:34.729200 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:34.729206 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:34.729234 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:35:34.729721 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:34.729739 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:34.731616 (+ 1877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:34.731624 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:34.733804 (+ 2180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:34.733821 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:34.736204 (+ 2383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:34.736211 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:34.736387 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:34.736418 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 18:35:34.736492 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 18:35:34.736638 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5051,"mutex_wait_us":80,"thread_start_us":4896,"threads_started":1}
I20260620 18:35:34.740433 23509 tablet_service.cc:1467] Processing CreateTablet for tablet 1c58d5ac67ea41309c5d6d8b6a1f7b37 (DEFAULT_TABLE table=test-table [id=1d3be0170b474699b56b463ea13a783b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:35:34.740432 23375 tablet_service.cc:1467] Processing CreateTablet for tablet 1c58d5ac67ea41309c5d6d8b6a1f7b37 (DEFAULT_TABLE table=test-table [id=1d3be0170b474699b56b463ea13a783b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:35:34.741719 23509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c58d5ac67ea41309c5d6d8b6a1f7b37. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:35:34.741748 23375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c58d5ac67ea41309c5d6d8b6a1f7b37. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:35:34.738361 23642 tablet_service.cc:1467] Processing CreateTablet for tablet 1c58d5ac67ea41309c5d6d8b6a1f7b37 (DEFAULT_TABLE table=test-table [id=1d3be0170b474699b56b463ea13a783b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:35:34.743312 23642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c58d5ac67ea41309c5d6d8b6a1f7b37. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:35:34.753057 23725 tablet_bootstrap.cc:492] T 1c58d5ac67ea41309c5d6d8b6a1f7b37 P 0ac8f29722034c799021207e3c6a9dcc: Bootstrap starting.
I20260620 18:35:34.756814 23725 tablet_bootstrap.cc:654] T 1c58d5ac67ea41309c5d6d8b6a1f7b37 P 0ac8f29722034c799021207e3c6a9dcc: Neither blocks nor log segments found. Creating new log.
I20260620 18:35:34.757143 23727 tablet_bootstrap.cc:492] T 1c58d5ac67ea41309c5d6d8b6a1f7b37 P 8001e68b4b3f4b06bd13ca50dad66fe8: Bootstrap starting.
I20260620 18:35:34.758201 23726 tablet_bootstrap.cc:492] T 1c58d5ac67ea41309c5d6d8b6a1f7b37 P b645921494d34e1ca613978ba9206b37: Bootstrap starting.
I20260620 18:35:34.759001 23725 log.cc:824] T 1c58d5ac67ea41309c5d6d8b6a1f7b37 P 0ac8f29722034c799021207e3c6a9dcc: Log is configured to *not* fsync() on all Append() calls
I20260620 18:35:34.760600 23727 tablet_bootstrap.cc:654] T 1c58d5ac67ea41309c5d6d8b6a1f7b37 P 8001e68b4b3f4b06bd13ca50dad66fe8: Neither blocks nor log segments found. Creating new log.
I20260620 18:35:34.761549 23726 tablet_bootstrap.cc:654] T 1c58d5ac67ea41309c5d6d8b6a1f7b37 P b645921494d34e1ca613978ba9206b37: Neither blocks nor log segments found. Creating new log.
I20260620 18:35:34.761868 23727 log.cc:824] T 1c58d5ac67ea41309c5d6d8b6a1f7b37 P 8001e68b4b3f4b06bd13ca50dad66fe8: Log is configured to *not* fsync() on all Append() calls
I20260620 18:35:34.762280 23725 tablet_bootstrap.cc:492] T 1c58d5ac67ea41309c5d6d8b6a1f7b37 P 0ac8f29722034c799021207e3c6a9dcc: No bootstrap required, opened a new log
I20260620 18:35:34.762602 23726 log.cc:824] T 1c58d5ac67ea41309c5d6d8b6a1f7b37 P b645921494d34e1ca613978ba9206b37: Log is configured to *not* fsync() on all Append() calls
I20260620 18:35:34.762604 23725 ts_tablet_manager.cc:1397] T 1c58d5ac67ea41309c5d6d8b6a1f7b37 P 0ac8f29722034c799021207e3c6a9dcc: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.006s
I20260620 18:35:34.765304 23726 tablet_bootstrap.cc:492] T 1c58d5ac67ea41309c5d6d8b6a1f7b37 P b645921494d34e1ca613978ba9206b37: No bootstrap required, opened a new log
I20260620 18:35:34.765584 23726 ts_tablet_manager.cc:1397] T 1c58d5ac67ea41309c5d6d8b6a1f7b37 P b645921494d34e1ca613978ba9206b37: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260620 18:35:34.766237 23727 tablet_bootstrap.cc:492] T 1c58d5ac67ea41309c5d6d8b6a1f7b37 P 8001e68b4b3f4b06bd13ca50dad66fe8: No bootstrap required, opened a new log
I20260620 18:35:34.766496 23727 ts_tablet_manager.cc:1397] T 1c58d5ac67ea41309c5d6d8b6a1f7b37 P 8001e68b4b3f4b06bd13ca50dad66fe8: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260620 18:35:34.772677 23726 raft_consensus.cc:348] T 1c58d5ac67ea41309c5d6d8b6a1f7b37 P b645921494d34e1ca613978ba9206b37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8001e68b4b3f4b06bd13ca50dad66fe8" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 46523 } } peers { permanent_uuid: "0ac8f29722034c799021207e3c6a9dcc" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 38349 } } peers { permanent_uuid: "b645921494d34e1ca613978ba9206b37" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 35027 } }
I20260620 18:35:34.773036 23726 raft_consensus.cc:374] T 1c58d5ac67ea41309c5d6d8b6a1f7b37 P b645921494d34e1ca613978ba9206b37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:35:34.773123 23726 raft_consensus.cc:729] T 1c58d5ac67ea41309c5d6d8b6a1f7b37 P b645921494d34e1ca613978ba9206b37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b645921494d34e1ca613978ba9206b37, State: Initialized, Role: FOLLOWER
I20260620 18:35:34.773116 23725 raft_consensus.cc:348] T 1c58d5ac67ea41309c5d6d8b6a1f7b37 P 0ac8f29722034c799021207e3c6a9dcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8001e68b4b3f4b06bd13ca50dad66fe8" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 46523 } } peers { permanent_uuid: "0ac8f29722034c799021207e3c6a9dcc" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 38349 } } peers { permanent_uuid: "b645921494d34e1ca613978ba9206b37" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 35027 } }
I20260620 18:35:34.773557 23725 raft_consensus.cc:374] T 1c58d5ac67ea41309c5d6d8b6a1f7b37 P 0ac8f29722034c799021207e3c6a9dcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:35:34.773656 23725 raft_consensus.cc:729] T 1c58d5ac67ea41309c5d6d8b6a1f7b37 P 0ac8f29722034c799021207e3c6a9dcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ac8f29722034c799021207e3c6a9dcc, State: Initialized, Role: FOLLOWER
I20260620 18:35:34.773804 23726 consensus_queue.cc:260] T 1c58d5ac67ea41309c5d6d8b6a1f7b37 P b645921494d34e1ca613978ba9206b37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8001e68b4b3f4b06bd13ca50dad66fe8" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 46523 } } peers { permanent_uuid: "0ac8f29722034c799021207e3c6a9dcc" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 38349 } } peers { permanent_uuid: "b645921494d34e1ca613978ba9206b37" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 35027 } }
I20260620 18:35:34.774209 23725 consensus_queue.cc:260] T 1c58d5ac67ea41309c5d6d8b6a1f7b37 P 0ac8f29722034c799021207e3c6a9dcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8001e68b4b3f4b06bd13ca50dad66fe8" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 46523 } } peers { permanent_uuid: "0ac8f29722034c799021207e3c6a9dcc" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 38349 } } peers { permanent_uuid: "b645921494d34e1ca613978ba9206b37" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 35027 } }
I20260620 18:35:34.777123 23726 ts_tablet_manager.cc:1428] T 1c58d5ac67ea41309c5d6d8b6a1f7b37 P b645921494d34e1ca613978ba9206b37: Time spent starting tablet: real 0.011s user 0.002s sys 0.009s
I20260620 18:35:34.777427 23727 raft_consensus.cc:348] T 1c58d5ac67ea41309c5d6d8b6a1f7b37 P 8001e68b4b3f4b06bd13ca50dad66fe8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8001e68b4b3f4b06bd13ca50dad66fe8" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 46523 } } peers { permanent_uuid: "0ac8f29722034c799021207e3c6a9dcc" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 38349 } } peers { permanent_uuid: "b645921494d34e1ca613978ba9206b37" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 35027 } }
I20260620 18:35:34.777949 23727 raft_consensus.cc:374] T 1c58d5ac67ea41309c5d6d8b6a1f7b37 P 8001e68b4b3f4b06bd13ca50dad66fe8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:35:34.778062 23727 raft_consensus.cc:729] T 1c58d5ac67ea41309c5d6d8b6a1f7b37 P 8001e68b4b3f4b06bd13ca50dad66fe8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8001e68b4b3f4b06bd13ca50dad66fe8, State: Initialized, Role: FOLLOWER
I20260620 18:35:34.778754 23727 consensus_queue.cc:260] T 1c58d5ac67ea41309c5d6d8b6a1f7b37 P 8001e68b4b3f4b06bd13ca50dad66fe8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8001e68b4b3f4b06bd13ca50dad66fe8" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 46523 } } peers { permanent_uuid: "0ac8f29722034c799021207e3c6a9dcc" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 38349 } } peers { permanent_uuid: "b645921494d34e1ca613978ba9206b37" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 35027 } }
I20260620 18:35:34.779932 23707 heartbeater.cc:503] Master 127.22.37.62:40775 was elected leader, sending a full tablet report...
I20260620 18:35:34.781940 23727 ts_tablet_manager.cc:1428] T 1c58d5ac67ea41309c5d6d8b6a1f7b37 P 8001e68b4b3f4b06bd13ca50dad66fe8: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20260620 18:35:34.782063 23725 ts_tablet_manager.cc:1428] T 1c58d5ac67ea41309c5d6d8b6a1f7b37 P 0ac8f29722034c799021207e3c6a9dcc: Time spent starting tablet: real 0.019s user 0.013s sys 0.006s
I20260620 18:35:34.782428 23574 heartbeater.cc:503] Master 127.22.37.62:40775 was elected leader, sending a full tablet report...
W20260620 18:35:34.891615 23575 tablet.cc:2369] T 1c58d5ac67ea41309c5d6d8b6a1f7b37 P 0ac8f29722034c799021207e3c6a9dcc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:35:34.892443 23708 tablet.cc:2369] T 1c58d5ac67ea41309c5d6d8b6a1f7b37 P 8001e68b4b3f4b06bd13ca50dad66fe8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:35:34.908218 23441 tablet.cc:2369] T 1c58d5ac67ea41309c5d6d8b6a1f7b37 P b645921494d34e1ca613978ba9206b37: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:35:34.942726 23733 raft_consensus.cc:482] T 1c58d5ac67ea41309c5d6d8b6a1f7b37 P 8001e68b4b3f4b06bd13ca50dad66fe8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:35:34.943078 23733 raft_consensus.cc:504] T 1c58d5ac67ea41309c5d6d8b6a1f7b37 P 8001e68b4b3f4b06bd13ca50dad66fe8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8001e68b4b3f4b06bd13ca50dad66fe8" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 46523 } } peers { permanent_uuid: "0ac8f29722034c799021207e3c6a9dcc" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 38349 } } peers { permanent_uuid: "b645921494d34e1ca613978ba9206b37" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 35027 } }
I20260620 18:35:34.944787 23733 leader_election.cc:288] T 1c58d5ac67ea41309c5d6d8b6a1f7b37 P 8001e68b4b3f4b06bd13ca50dad66fe8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ac8f29722034c799021207e3c6a9dcc (127.22.37.2:38349), b645921494d34e1ca613978ba9206b37 (127.22.37.1:35027)
I20260620 18:35:34.951251 23709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:34.945132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.1:35027 (local address 127.22.37.3:58515)
0620 18:35:34.945435 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:34.945454 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:34.945581 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:34.946002 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:34.946014 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:34.946048 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:35:34.946254 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:34.946270 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:34.947504 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:34.947507 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:34.949417 (+ 1910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:34.949431 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:34.950743 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:34.950747 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:34.950868 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:34.950891 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 18:35:34.950959 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 18:35:34.951046 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":18}
I20260620 18:35:34.951428 23721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:34.945238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.3:58515 (local address 127.22.37.1:35027)
0620 18:35:34.945451 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:34.945459 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:34.945547 (+ 88us) server_negotiation.cc:407] Connection header received
0620 18:35:34.945730 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:34.945739 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:34.945841 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:34.945943 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:35:34.946414 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:34.947390 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:34.949568 (+ 2178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:34.950583 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:34.950714 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:34.950815 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:34.951009 (+ 194us) server_negotiation.cc:299] Negotiation successful
0620 18:35:34.951143 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 18:35:34.952031 23734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:34.945149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.2:38349 (local address 127.22.37.3:33303)
0620 18:35:34.946168 (+ 1019us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:34.946195 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:34.946341 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:34.946648 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:34.946651 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:34.946669 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:35:34.946931 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:34.946938 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:34.948018 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:34.948022 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:34.949879 (+ 1857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:34.949889 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:34.951389 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:34.951397 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:34.951545 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:34.951578 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 18:35:34.951670 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 18:35:34.951778 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":909,"mutex_wait_us":276,"thread_start_us":431,"threads_started":1}
I20260620 18:35:34.952265 23724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:34.945211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.3:33303 (local address 127.22.37.2:38349)
0620 18:35:34.945439 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:34.945446 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:34.946307 (+ 861us) server_negotiation.cc:407] Connection header received
0620 18:35:34.946465 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:34.946472 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:34.946546 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:34.946642 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:35:34.947072 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:34.947891 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:34.950013 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:34.951193 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:34.951325 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:34.951411 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:34.951872 (+ 461us) server_negotiation.cc:299] Negotiation successful
0620 18:35:34.952006 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 18:35:34.952584 23395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c58d5ac67ea41309c5d6d8b6a1f7b37" candidate_uuid: "8001e68b4b3f4b06bd13ca50dad66fe8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b645921494d34e1ca613978ba9206b37" is_pre_election: true
I20260620 18:35:34.953092 23395 raft_consensus.cc:2454] T 1c58d5ac67ea41309c5d6d8b6a1f7b37 P b645921494d34e1ca613978ba9206b37 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8001e68b4b3f4b06bd13ca50dad66fe8 in term 0.
I20260620 18:35:34.953109 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c58d5ac67ea41309c5d6d8b6a1f7b37" candidate_uuid: "8001e68b4b3f4b06bd13ca50dad66fe8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ac8f29722034c799021207e3c6a9dcc" is_pre_election: true
I20260620 18:35:34.953647 23529 raft_consensus.cc:2454] T 1c58d5ac67ea41309c5d6d8b6a1f7b37 P 0ac8f29722034c799021207e3c6a9dcc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8001e68b4b3f4b06bd13ca50dad66fe8 in term 0.
I20260620 18:35:34.953815 23595 leader_election.cc:302] T 1c58d5ac67ea41309c5d6d8b6a1f7b37 P 8001e68b4b3f4b06bd13ca50dad66fe8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8001e68b4b3f4b06bd13ca50dad66fe8, b645921494d34e1ca613978ba9206b37; no voters:
I20260620 18:35:34.954226 23733 raft_consensus.cc:2781] T 1c58d5ac67ea41309c5d6d8b6a1f7b37 P 8001e68b4b3f4b06bd13ca50dad66fe8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:35:34.954378 23733 raft_consensus.cc:482] T 1c58d5ac67ea41309c5d6d8b6a1f7b37 P 8001e68b4b3f4b06bd13ca50dad66fe8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:35:34.954485 23733 raft_consensus.cc:3037] T 1c58d5ac67ea41309c5d6d8b6a1f7b37 P 8001e68b4b3f4b06bd13ca50dad66fe8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:35:34.956599 23733 raft_consensus.cc:504] T 1c58d5ac67ea41309c5d6d8b6a1f7b37 P 8001e68b4b3f4b06bd13ca50dad66fe8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8001e68b4b3f4b06bd13ca50dad66fe8" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 46523 } } peers { permanent_uuid: "0ac8f29722034c799021207e3c6a9dcc" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 38349 } } peers { permanent_uuid: "b645921494d34e1ca613978ba9206b37" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 35027 } }
I20260620 18:35:34.957202 23733 leader_election.cc:288] T 1c58d5ac67ea41309c5d6d8b6a1f7b37 P 8001e68b4b3f4b06bd13ca50dad66fe8 [CANDIDATE]: Term 1 election: Requested vote from peers 0ac8f29722034c799021207e3c6a9dcc (127.22.37.2:38349), b645921494d34e1ca613978ba9206b37 (127.22.37.1:35027)
I20260620 18:35:34.957618 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c58d5ac67ea41309c5d6d8b6a1f7b37" candidate_uuid: "8001e68b4b3f4b06bd13ca50dad66fe8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ac8f29722034c799021207e3c6a9dcc"
I20260620 18:35:34.957706 23395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c58d5ac67ea41309c5d6d8b6a1f7b37" candidate_uuid: "8001e68b4b3f4b06bd13ca50dad66fe8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b645921494d34e1ca613978ba9206b37"
I20260620 18:35:34.957839 23529 raft_consensus.cc:3037] T 1c58d5ac67ea41309c5d6d8b6a1f7b37 P 0ac8f29722034c799021207e3c6a9dcc [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:35:34.957930 23395 raft_consensus.cc:3037] T 1c58d5ac67ea41309c5d6d8b6a1f7b37 P b645921494d34e1ca613978ba9206b37 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:35:34.960999 23395 raft_consensus.cc:2454] T 1c58d5ac67ea41309c5d6d8b6a1f7b37 P b645921494d34e1ca613978ba9206b37 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8001e68b4b3f4b06bd13ca50dad66fe8 in term 1.
I20260620 18:35:34.960999 23529 raft_consensus.cc:2454] T 1c58d5ac67ea41309c5d6d8b6a1f7b37 P 0ac8f29722034c799021207e3c6a9dcc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8001e68b4b3f4b06bd13ca50dad66fe8 in term 1.
I20260620 18:35:34.961410 23596 leader_election.cc:302] T 1c58d5ac67ea41309c5d6d8b6a1f7b37 P 8001e68b4b3f4b06bd13ca50dad66fe8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ac8f29722034c799021207e3c6a9dcc, 8001e68b4b3f4b06bd13ca50dad66fe8; no voters:
I20260620 18:35:34.961691 23733 raft_consensus.cc:2781] T 1c58d5ac67ea41309c5d6d8b6a1f7b37 P 8001e68b4b3f4b06bd13ca50dad66fe8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:35:34.962107 23733 raft_consensus.cc:686] T 1c58d5ac67ea41309c5d6d8b6a1f7b37 P 8001e68b4b3f4b06bd13ca50dad66fe8 [term 1 LEADER]: Becoming Leader. State: Replica: 8001e68b4b3f4b06bd13ca50dad66fe8, State: Running, Role: LEADER
I20260620 18:35:34.962598 23733 consensus_queue.cc:237] T 1c58d5ac67ea41309c5d6d8b6a1f7b37 P 8001e68b4b3f4b06bd13ca50dad66fe8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8001e68b4b3f4b06bd13ca50dad66fe8" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 46523 } } peers { permanent_uuid: "0ac8f29722034c799021207e3c6a9dcc" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 38349 } } peers { permanent_uuid: "b645921494d34e1ca613978ba9206b37" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 35027 } }
I20260620 18:35:34.967131 23250 catalog_manager.cc:5495] T 1c58d5ac67ea41309c5d6d8b6a1f7b37 P 8001e68b4b3f4b06bd13ca50dad66fe8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8001e68b4b3f4b06bd13ca50dad66fe8 (127.22.37.3). New cstate: current_term: 1 leader_uuid: "8001e68b4b3f4b06bd13ca50dad66fe8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8001e68b4b3f4b06bd13ca50dad66fe8" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 46523 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ac8f29722034c799021207e3c6a9dcc" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 38349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b645921494d34e1ca613978ba9206b37" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 35027 } health_report { overall_health: UNKNOWN } } }
Jun 20 18:35:35 dist-test-slave-5lgp krb5kdc[23206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.37.1@KRBTEST.COM
I20260620 18:35:35.035867 23721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:35.020966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37774 (local address 127.22.37.1:35027)
0620 18:35:35.021274 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:35.021281 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:35.021522 (+ 241us) server_negotiation.cc:407] Connection header received
0620 18:35:35.021674 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:35.021677 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:35.021785 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:35.021893 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:35.023367 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:35.024205 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:35.025646 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:35.026099 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:35.026208 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:35.030397 (+ 4189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:35.030414 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:35.030431 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:35.030473 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:35.033260 (+ 2787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:35.033983 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:35.033991 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:35.033993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:35.034135 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:35.034478 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:35.034481 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:35.034483 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:35.034840 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:35.035054 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:35.035317 (+ 263us) server_negotiation.cc:299] Negotiation successful
0620 18:35:35.035480 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
Jun 20 18:35:35 dist-test-slave-5lgp krb5kdc[23206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980535, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 18:35:35 dist-test-slave-5lgp krb5kdc[23206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980535, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.22.37.1@KRBTEST.COM
I20260620 18:35:35.084234 23721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:35.068827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37780 (local address 127.22.37.1:35027)
0620 18:35:35.069110 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:35.069117 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:35.069335 (+ 218us) server_negotiation.cc:407] Connection header received
0620 18:35:35.069606 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:35.069611 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:35.069694 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:35.069785 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:35.071127 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:35.072059 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:35.073295 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:35.074005 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:35.074129 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:35.078259 (+ 4130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:35.078266 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:35.078271 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:35.078319 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:35.081733 (+ 3414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:35.082416 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:35.082421 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:35.082425 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:35.082575 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:35.082936 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:35.082940 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:35.082944 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:35.083269 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:35.083541 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:35.083682 (+ 141us) server_negotiation.cc:299] Negotiation successful
0620 18:35:35.083894 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132}
Jun 20 18:35:35 dist-test-slave-5lgp krb5kdc[23206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980535, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 18:35:35 dist-test-slave-5lgp krb5kdc[23206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980535, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.37.1@KRBTEST.COM
I20260620 18:35:35.129772 23721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:35.115676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37786 (local address 127.22.37.1:35027)
0620 18:35:35.115952 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:35.115959 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:35.116285 (+ 326us) server_negotiation.cc:407] Connection header received
0620 18:35:35.116488 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:35.116494 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:35.116560 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:35.116664 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:35.118117 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:35.118813 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:35.120011 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:35.120438 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:35.120567 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:35.124538 (+ 3971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:35.124543 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:35.124546 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:35.124583 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:35.127049 (+ 2466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:35.127701 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:35.127707 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:35.127711 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:35.127832 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:35.128282 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:35.128287 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:35.128295 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:35.128581 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:35.128843 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:35.129133 (+ 290us) server_negotiation.cc:299] Negotiation successful
0620 18:35:35.129412 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260620 18:35:35.131765 23355 tablet_service.cc:2908] Rejecting scan request for tablet 1c58d5ac67ea41309c5d6d8b6a1f7b37: Uninitialized: clean time has not yet been initialized
I20260620 18:35:35.134735 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 23312
I20260620 18:35:35.152726 23721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:35.136354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37796 (local address 127.22.37.1:35027)
0620 18:35:35.136618 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:35.136627 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:35.137056 (+ 429us) server_negotiation.cc:407] Connection header received
0620 18:35:35.137255 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:35.137262 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:35.137402 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:35.137516 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:35.139374 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:35.140626 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:35.142477 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:35.143016 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:35.143105 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:35.146885 (+ 3780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:35.146892 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:35.146898 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:35.146937 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:35.150116 (+ 3179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:35.150756 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:35.150762 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:35.150765 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:35.150913 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:35.151249 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:35.151254 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:35.151258 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:35.151598 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:35.151880 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:35.152153 (+ 273us) server_negotiation.cc:299] Negotiation successful
0620 18:35:35.152387 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 18:35:35.153344 23435 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:35:35.241989 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 23312
I20260620 18:35:35.258687 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 23446
Jun 20 18:35:35 dist-test-slave-5lgp krb5kdc[23206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980535, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.37.2@KRBTEST.COM
I20260620 18:35:35.272783 23724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:35.259401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50076 (local address 127.22.37.2:38349)
0620 18:35:35.259558 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:35.259562 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:35.259581 (+ 19us) server_negotiation.cc:407] Connection header received
0620 18:35:35.259690 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:35.259694 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:35.259755 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:35.259838 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:35.260994 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:35.261784 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:35.263024 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:35.263433 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:35.263535 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:35.267431 (+ 3896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:35.267447 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:35.267464 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:35.267509 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:35.270333 (+ 2824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:35.270951 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:35.270959 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:35.270961 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:35.271086 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:35.271423 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:35.271427 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:35.271429 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:35.271825 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:35.272058 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:35.272363 (+ 305us) server_negotiation.cc:299] Negotiation successful
0620 18:35:35.272524 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 18:35:35.273502 23569 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:35:35.360700 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 23446
I20260620 18:35:35.377043 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 23579
Jun 20 18:35:35 dist-test-slave-5lgp krb5kdc[23206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980535, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.37.3@KRBTEST.COM
I20260620 18:35:35.391554 23750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:35.377829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55970 (local address 127.22.37.3:46523)
0620 18:35:35.378304 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:35.378309 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:35.378324 (+ 15us) server_negotiation.cc:407] Connection header received
0620 18:35:35.378377 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:35.378380 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:35.378442 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:35.378557 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:35.379766 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:35.380432 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:35.381854 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:35.382325 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:35.382454 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:35.386445 (+ 3991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:35.386461 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:35.386477 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:35.386511 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:35.389109 (+ 2598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:35.389827 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:35.389834 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:35.389837 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:35.389980 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:35.390342 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:35.390345 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:35.390347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:35.390679 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:35.390884 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:35.391076 (+ 192us) server_negotiation.cc:299] Negotiation successful
0620 18:35:35.391234 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":212,"threads_started":1}
I20260620 18:35:35.392206 23702 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:35:35.481530 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 23579
I20260620 18:35:35.498271 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 23219
I20260620 18:35:35.499400 23281 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:35:35.606840 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 23219
2026-06-20T18:35:35Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4205 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980527431848-22676-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:35:35 dist-test-slave-5lgp krb5kdc[23756](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:35:35 dist-test-slave-5lgp krb5kdc[23756](info): set up 2 sockets
Jun 20 18:35:35 dist-test-slave-5lgp krb5kdc[23756](info): commencing operation
krb5kdc: starting...
W20260620 18:35:36.689301 22676 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.001s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980527431848-22676-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980527431848-22676-0/krb5kdc/test-user.keytab.
Jun 20 18:35:36 dist-test-slave-5lgp krb5kdc[23756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980536, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:35:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:35:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.37.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.37.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.37.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab.
Entry for principal kudu/127.22.37.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab.
I20260620 18:35:36.872684 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980527431848-22676-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980527431848-22676-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980527431848-22676-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.62:37885
--webserver_interface=127.22.37.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab
--principal=kudu/127.22.37.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:45519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.37.62:37885
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:35:37.197868 23769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:35:37.198205 23769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:35:37.198379 23769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:35:37.205559 23769 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:35:37.205667 23769 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 18:35:37.205754 23769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:35:37.205842 23769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:35:37.205932 23769 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:35:37.206032 23769 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:35:37.215888 23769 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:45519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980527431848-22676-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980527431848-22676-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.37.62:37885
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab
--principal=kudu/127.22.37.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.62:37885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.37.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980527431848-22676-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:35:37.217216 23769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:35:37.218699 23769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:35:37.230082 23774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:37.230160 23777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:37.230132 23775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:35:37.231153 23769 server_base.cc:1034] running on GCE node
I20260620 18:35:37.231861 23769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:35:37.233238 23769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:35:37.234474 23769 hybrid_clock.cc:648] HybridClock initialized: now 1781980537234416 us; error 39 us; skew 500 ppm
I20260620 18:35:37.234771 23769 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:35:37 dist-test-slave-5lgp krb5kdc[23756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980537, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:35:37.237843 23769 init.cc:377] Logged in from keytab as kudu/127.22.37.62@KRBTEST.COM (short username kudu)
I20260620 18:35:37.239181 23769 webserver.cc:460] Webserver started at http://127.22.37.62:43573/ using document root <none> and password file <none>
I20260620 18:35:37.239813 23769 fs_manager.cc:359] Metadata directory not provided
I20260620 18:35:37.239902 23769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:35:37.240180 23769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:35:37.243067 23769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980527431848-22676-0/minicluster-data/master-0/data/instance:
uuid: "f72af03185aa4be6bfacf552ced6b7a4"
format_stamp: "Formatted at 2026-06-20 18:35:37 on dist-test-slave-5lgp"
server_key: "77d24d98e178f50484cc95d309461b9a"
server_key_iv: "759dee6fb8ea41ea5c82a1027b63e7c4"
server_key_version: "encryptionkey@0"
I20260620 18:35:37.243800 23769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980527431848-22676-0/minicluster-data/master-0/wal/instance:
uuid: "f72af03185aa4be6bfacf552ced6b7a4"
format_stamp: "Formatted at 2026-06-20 18:35:37 on dist-test-slave-5lgp"
server_key: "77d24d98e178f50484cc95d309461b9a"
server_key_iv: "759dee6fb8ea41ea5c82a1027b63e7c4"
server_key_version: "encryptionkey@0"
I20260620 18:35:37.249143 23769 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 18:35:37.252645 23784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:37.254336 23769 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:35:37.254477 23769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980527431848-22676-0/minicluster-data/master-0/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980527431848-22676-0/minicluster-data/master-0/wal
uuid: "f72af03185aa4be6bfacf552ced6b7a4"
format_stamp: "Formatted at 2026-06-20 18:35:37 on dist-test-slave-5lgp"
server_key: "77d24d98e178f50484cc95d309461b9a"
server_key_iv: "759dee6fb8ea41ea5c82a1027b63e7c4"
server_key_version: "encryptionkey@0"
I20260620 18:35:37.254663 23769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980527431848-22676-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980527431848-22676-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980527431848-22676-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:35:37.274621 23769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:35:37.278585 23769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:35:37.278812 23769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:35:37.298995 23769 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.62:37885
I20260620 18:35:37.299005 23845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.62:37885 every 8 connection(s)
I20260620 18:35:37.300348 23769 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
I20260620 18:35:37.304204 23846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:35:37.310078 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 23769
I20260620 18:35:37.310314 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980527431848-22676-0/minicluster-data/master-0/wal/instance
I20260620 18:35:37.310951 22676 external_mini_cluster.cc:1428] Setting key 5df867b2cb52df2eaee6bff9236c31b0
I20260620 18:35:37.316243 23846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f72af03185aa4be6bfacf552ced6b7a4: Bootstrap starting.
I20260620 18:35:37.320030 23846 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f72af03185aa4be6bfacf552ced6b7a4: Neither blocks nor log segments found. Creating new log.
I20260620 18:35:37.321135 23846 log.cc:824] T 00000000000000000000000000000000 P f72af03185aa4be6bfacf552ced6b7a4: Log is configured to *not* fsync() on all Append() calls
I20260620 18:35:37.324223 23846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f72af03185aa4be6bfacf552ced6b7a4: No bootstrap required, opened a new log
Jun 20 18:35:37 dist-test-slave-5lgp krb5kdc[23756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980536, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM
I20260620 18:35:37.334379 23846 raft_consensus.cc:348] T 00000000000000000000000000000000 P f72af03185aa4be6bfacf552ced6b7a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f72af03185aa4be6bfacf552ced6b7a4" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 37885 } }
I20260620 18:35:37.334830 23846 raft_consensus.cc:374] T 00000000000000000000000000000000 P f72af03185aa4be6bfacf552ced6b7a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:35:37.334939 23846 raft_consensus.cc:729] T 00000000000000000000000000000000 P f72af03185aa4be6bfacf552ced6b7a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f72af03185aa4be6bfacf552ced6b7a4, State: Initialized, Role: FOLLOWER
I20260620 18:35:37.335410 23849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:37.312957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47132 (local address 127.22.37.62:37885)
0620 18:35:37.313822 (+ 865us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:37.313841 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:37.313893 (+ 52us) server_negotiation.cc:407] Connection header received
0620 18:35:37.316244 (+ 2351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:37.316274 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:37.316770 (+ 496us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:37.317209 (+ 439us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:37.318535 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:37.320522 (+ 1987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:37.321918 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:37.322755 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:37.322930 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:37.326792 (+ 3862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:37.326802 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:37.326821 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:37.326874 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:37.330863 (+ 3989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:37.331479 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:37.331488 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:37.331497 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:37.331722 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:37.332157 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:37.332162 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:37.332166 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:37.332919 (+ 753us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:37.333223 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:37.333614 (+ 391us) server_negotiation.cc:299] Negotiation successful
0620 18:35:37.334050 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":373,"threads_started":1}
I20260620 18:35:37.335621 23846 consensus_queue.cc:260] T 00000000000000000000000000000000 P f72af03185aa4be6bfacf552ced6b7a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f72af03185aa4be6bfacf552ced6b7a4" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 37885 } }
I20260620 18:35:37.335906 23846 raft_consensus.cc:388] T 00000000000000000000000000000000 P f72af03185aa4be6bfacf552ced6b7a4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:35:37.336066 23846 raft_consensus.cc:482] T 00000000000000000000000000000000 P f72af03185aa4be6bfacf552ced6b7a4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:35:37.336253 23846 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f72af03185aa4be6bfacf552ced6b7a4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:35:37.338771 23846 raft_consensus.cc:504] T 00000000000000000000000000000000 P f72af03185aa4be6bfacf552ced6b7a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f72af03185aa4be6bfacf552ced6b7a4" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 37885 } }
I20260620 18:35:37.339249 23846 leader_election.cc:302] T 00000000000000000000000000000000 P f72af03185aa4be6bfacf552ced6b7a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f72af03185aa4be6bfacf552ced6b7a4; no voters:
I20260620 18:35:37.339828 23846 leader_election.cc:288] T 00000000000000000000000000000000 P f72af03185aa4be6bfacf552ced6b7a4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:35:37.339953 23851 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f72af03185aa4be6bfacf552ced6b7a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:35:37.340721 23851 raft_consensus.cc:686] T 00000000000000000000000000000000 P f72af03185aa4be6bfacf552ced6b7a4 [term 1 LEADER]: Becoming Leader. State: Replica: f72af03185aa4be6bfacf552ced6b7a4, State: Running, Role: LEADER
I20260620 18:35:37.341257 23851 consensus_queue.cc:237] T 00000000000000000000000000000000 P f72af03185aa4be6bfacf552ced6b7a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f72af03185aa4be6bfacf552ced6b7a4" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 37885 } }
I20260620 18:35:37.341892 23846 sys_catalog.cc:564] T 00000000000000000000000000000000 P f72af03185aa4be6bfacf552ced6b7a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:35:37.343788 23852 sys_catalog.cc:455] T 00000000000000000000000000000000 P f72af03185aa4be6bfacf552ced6b7a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f72af03185aa4be6bfacf552ced6b7a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f72af03185aa4be6bfacf552ced6b7a4" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 37885 } } }
I20260620 18:35:37.344235 23852 sys_catalog.cc:458] T 00000000000000000000000000000000 P f72af03185aa4be6bfacf552ced6b7a4 [sys.catalog]: This master's current role is: LEADER
I20260620 18:35:37.344138 23853 sys_catalog.cc:455] T 00000000000000000000000000000000 P f72af03185aa4be6bfacf552ced6b7a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f72af03185aa4be6bfacf552ced6b7a4. Latest consensus state: current_term: 1 leader_uuid: "f72af03185aa4be6bfacf552ced6b7a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f72af03185aa4be6bfacf552ced6b7a4" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 37885 } } }
I20260620 18:35:37.344412 23853 sys_catalog.cc:458] T 00000000000000000000000000000000 P f72af03185aa4be6bfacf552ced6b7a4 [sys.catalog]: This master's current role is: LEADER
I20260620 18:35:37.353885 23856 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:35:37.358301 23856 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:35:37.366503 23856 catalog_manager.cc:1347] Generated new cluster ID: 6a9a40adf7af43acabf49c4d6bd7909a
I20260620 18:35:37.366628 23856 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:35:37.390040 23856 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:35:37.391366 23856 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:35:37.397662 23856 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f72af03185aa4be6bfacf552ced6b7a4: Generated new TSK 0
I20260620 18:35:37.398459 23856 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.37.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.37.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.37.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.1.keytab.
Entry for principal kudu/127.22.37.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.1.keytab.
I20260620 18:35:37.453419 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980527431848-22676-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980527431848-22676-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980527431848-22676-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980527431848-22676-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.37.1:0
--local_ip_for_outbound_sockets=127.22.37.1
--webserver_interface=127.22.37.1
--webserver_port=0
--tserver_master_addrs=127.22.37.62:37885
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.1.keytab
--principal=kudu/127.22.37.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:45519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:35:37.785212 23872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:35:37.785542 23872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:35:37.785713 23872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:35:37.792590 23872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:35:37.792743 23872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:35:37.792912 23872 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 18:35:37.793053 23872 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.37.1
I20260620 18:35:37.802374 23872 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:45519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980527431848-22676-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980527431848-22676-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.1.keytab
--principal=kudu/127.22.37.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.37.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980527431848-22676-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.37.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.37.62:37885
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.37.1
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980527431848-22676-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:35:37.803665 23872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:35:37.804986 23872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:35:37.817946 23877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:37.817965 23880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:37.817952 23878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:35:37.818744 23872 server_base.cc:1034] running on GCE node
I20260620 18:35:37.819344 23872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:35:37.820272 23872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:35:37.821481 23872 hybrid_clock.cc:648] HybridClock initialized: now 1781980537821404 us; error 48 us; skew 500 ppm
I20260620 18:35:37.821777 23872 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:35:37 dist-test-slave-5lgp krb5kdc[23756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980537, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:35:37.824797 23872 init.cc:377] Logged in from keytab as kudu/127.22.37.1@KRBTEST.COM (short username kudu)
I20260620 18:35:37.826287 23872 webserver.cc:460] Webserver started at http://127.22.37.1:38073/ using document root <none> and password file <none>
I20260620 18:35:37.826963 23872 fs_manager.cc:359] Metadata directory not provided
I20260620 18:35:37.827087 23872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:35:37.827412 23872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:35:37.830415 23872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980527431848-22676-0/minicluster-data/ts-0/data/instance:
uuid: "2fdb54ee16854fcbbb83f8a380d57e5d"
format_stamp: "Formatted at 2026-06-20 18:35:37 on dist-test-slave-5lgp"
server_key: "c0850edd6ddc1fbd08adee6b1a17f335"
server_key_iv: "6ff799eda6d6f53d0a983df03e4ec743"
server_key_version: "encryptionkey@0"
I20260620 18:35:37.831246 23872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980527431848-22676-0/minicluster-data/ts-0/wal/instance:
uuid: "2fdb54ee16854fcbbb83f8a380d57e5d"
format_stamp: "Formatted at 2026-06-20 18:35:37 on dist-test-slave-5lgp"
server_key: "c0850edd6ddc1fbd08adee6b1a17f335"
server_key_iv: "6ff799eda6d6f53d0a983df03e4ec743"
server_key_version: "encryptionkey@0"
I20260620 18:35:37.836248 23872 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 18:35:37.839507 23887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:37.840780 23872 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260620 18:35:37.840941 23872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980527431848-22676-0/minicluster-data/ts-0/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980527431848-22676-0/minicluster-data/ts-0/wal
uuid: "2fdb54ee16854fcbbb83f8a380d57e5d"
format_stamp: "Formatted at 2026-06-20 18:35:37 on dist-test-slave-5lgp"
server_key: "c0850edd6ddc1fbd08adee6b1a17f335"
server_key_iv: "6ff799eda6d6f53d0a983df03e4ec743"
server_key_version: "encryptionkey@0"
I20260620 18:35:37.841148 23872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980527431848-22676-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980527431848-22676-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980527431848-22676-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:35:37.868523 23872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:35:37.872917 23872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:35:37.873204 23872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:35:37.875741 23872 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:35:37.875846 23872 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:37.875998 23872 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:35:37.876070 23872 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 18:35:37 dist-test-slave-5lgp krb5kdc[23756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980537, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.1@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM
I20260620 18:35:37.898248 23896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:37.876451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.1:44727 (local address 127.22.37.62:37885)
0620 18:35:37.876990 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:37.876997 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:37.878234 (+ 1237us) server_negotiation.cc:407] Connection header received
0620 18:35:37.880224 (+ 1990us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:37.880230 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:37.880317 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:37.880432 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:37.883270 (+ 2838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:37.884196 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:37.887457 (+ 3261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:37.888023 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:37.888148 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:37.892751 (+ 4603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:37.892756 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:37.892759 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:37.892790 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:37.895161 (+ 2371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:37.895805 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:37.895809 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:37.895813 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:37.895923 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:37.896324 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:37.896329 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:37.896332 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:37.896589 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:37.896752 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:37.897628 (+ 876us) server_negotiation.cc:299] Negotiation successful
0620 18:35:37.897860 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":311,"threads_started":1}
I20260620 18:35:37.899272 23897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:37.877129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:37885 (local address 127.22.37.1:44727)
0620 18:35:37.878059 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:37.878108 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:37.879947 (+ 1839us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:37.880924 (+ 977us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:37.880938 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:37.881957 (+ 1019us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:37.883039 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:37.883076 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:37.884371 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:37.884379 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:37.887263 (+ 2884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:37.887286 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:37.888461 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:37.888470 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:37.888793 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:37.889652 (+ 859us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:35:37.889690 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:37.892501 (+ 2811us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:35:37.895290 (+ 2789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:37.895299 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:37.895313 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:37.895688 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:37.896023 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:37.896028 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:37.896032 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:37.896217 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:37.896715 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:35:37.896722 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:35:37.897122 (+ 400us) client_negotiation.cc:769] Sending connection context
0620 18:35:37.897542 (+ 420us) client_negotiation.cc:240] Negotiation successful
0620 18:35:37.898154 (+ 612us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":669,"thread_start_us":440,"threads_started":1}
I20260620 18:35:37.917783 23896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:37.905187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.1:55587 (local address 127.22.37.62:37885)
0620 18:35:37.905325 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:37.905328 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:37.906046 (+ 718us) server_negotiation.cc:407] Connection header received
0620 18:35:37.906245 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:37.906249 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:37.906306 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:37.906398 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:37.907246 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:37.907918 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:37.909185 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:37.909801 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:37.909950 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:37.911756 (+ 1806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:37.911763 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:37.911768 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:37.911807 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:37.914362 (+ 2555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:37.915385 (+ 1023us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:37.915391 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:37.915394 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:37.915548 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:37.916016 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:37.916021 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:37.916024 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:37.916426 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:37.916598 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:37.917183 (+ 585us) server_negotiation.cc:299] Negotiation successful
0620 18:35:37.917433 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 18:35:37.917985 23950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:37.905321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:37885 (local address 127.22.37.1:55587)
0620 18:35:37.905949 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:37.905967 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:37.906107 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:37.906418 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:37.906422 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:37.906842 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:37.907110 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:37.907117 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:37.908036 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:37.908043 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:37.909041 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:37.909051 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:37.910055 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:37.910062 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:37.910227 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:37.910896 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:35:37.910912 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:37.911587 (+ 675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:35:37.914500 (+ 2913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:37.914511 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:37.914517 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:37.915232 (+ 715us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:37.915682 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:37.915692 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:37.915695 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:37.915889 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:37.916590 (+ 701us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:35:37.916596 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:35:37.916883 (+ 287us) client_negotiation.cc:769] Sending connection context
0620 18:35:37.917087 (+ 204us) client_negotiation.cc:240] Negotiation successful
0620 18:35:37.917452 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":356,"threads_started":1}
I20260620 18:35:37.934147 23872 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.1:44765
I20260620 18:35:37.934170 24007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.1:44765 every 8 connection(s)
I20260620 18:35:37.935453 23872 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980527431848-22676-0/minicluster-data/ts-0/data/info.pb
I20260620 18:35:37.937767 24008 heartbeater.cc:347] Connected to a master server at 127.22.37.62:37885
I20260620 18:35:37.937983 24008 heartbeater.cc:464] Registering TS with master...
I20260620 18:35:37.938652 24008 heartbeater.cc:511] Master 127.22.37.62:37885 requested a full tablet report, sending...
I20260620 18:35:37.940600 23801 ts_manager.cc:194] Registered new tserver with Master: 2fdb54ee16854fcbbb83f8a380d57e5d (127.22.37.1:44765)
I20260620 18:35:37.942461 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 23872
I20260620 18:35:37.942569 23801 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.37.1@KRBTEST.COM'} at 127.22.37.1:44727
I20260620 18:35:37.942698 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980527431848-22676-0/minicluster-data/ts-0/wal/instance
I20260620 18:35:37.943389 22676 external_mini_cluster.cc:1428] Setting key eaaf24f747f635972287c441303dd91f
WARNING: no policy specified for kudu/127.22.37.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.37.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.37.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.2.keytab.
Entry for principal kudu/127.22.37.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.2.keytab.
I20260620 18:35:37.985512 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980527431848-22676-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980527431848-22676-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980527431848-22676-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980527431848-22676-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.37.2:0
--local_ip_for_outbound_sockets=127.22.37.2
--webserver_interface=127.22.37.2
--webserver_port=0
--tserver_master_addrs=127.22.37.62:37885
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.2.keytab
--principal=kudu/127.22.37.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:45519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:35:38.300997 24012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:35:38.301347 24012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:35:38.301515 24012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:35:38.312976 24012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:35:38.313109 24012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:35:38.313227 24012 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 18:35:38.313361 24012 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.37.2
I20260620 18:35:38.326304 24012 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:45519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980527431848-22676-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980527431848-22676-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.2.keytab
--principal=kudu/127.22.37.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.37.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980527431848-22676-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.37.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.37.62:37885
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.37.2
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980527431848-22676-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:35:38.327504 24012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:35:38.328780 24012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:35:38.341682 24017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:38.341832 24018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:38.341883 24020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:35:38.342748 24012 server_base.cc:1034] running on GCE node
I20260620 18:35:38.343395 24012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:35:38.344249 24012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:35:38.345508 24012 hybrid_clock.cc:648] HybridClock initialized: now 1781980538345427 us; error 41 us; skew 500 ppm
I20260620 18:35:38.345799 24012 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:35:38 dist-test-slave-5lgp krb5kdc[23756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980538, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:35:38.348721 24012 init.cc:377] Logged in from keytab as kudu/127.22.37.2@KRBTEST.COM (short username kudu)
I20260620 18:35:38.350152 24012 webserver.cc:460] Webserver started at http://127.22.37.2:33763/ using document root <none> and password file <none>
I20260620 18:35:38.350824 24012 fs_manager.cc:359] Metadata directory not provided
I20260620 18:35:38.350946 24012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:35:38.351248 24012 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:35:38.354261 24012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980527431848-22676-0/minicluster-data/ts-1/data/instance:
uuid: "82ee0b3ce78a4e2c996a42b9644afd48"
format_stamp: "Formatted at 2026-06-20 18:35:38 on dist-test-slave-5lgp"
server_key: "fecce8fc5754787cde6cf97c6c3f75f0"
server_key_iv: "51a7fe792ff7ce7a924d2812bb5519f3"
server_key_version: "encryptionkey@0"
I20260620 18:35:38.355037 24012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980527431848-22676-0/minicluster-data/ts-1/wal/instance:
uuid: "82ee0b3ce78a4e2c996a42b9644afd48"
format_stamp: "Formatted at 2026-06-20 18:35:38 on dist-test-slave-5lgp"
server_key: "fecce8fc5754787cde6cf97c6c3f75f0"
server_key_iv: "51a7fe792ff7ce7a924d2812bb5519f3"
server_key_version: "encryptionkey@0"
I20260620 18:35:38.360010 24012 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 18:35:38.363555 24027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:38.364869 24012 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:35:38.365046 24012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980527431848-22676-0/minicluster-data/ts-1/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980527431848-22676-0/minicluster-data/ts-1/wal
uuid: "82ee0b3ce78a4e2c996a42b9644afd48"
format_stamp: "Formatted at 2026-06-20 18:35:38 on dist-test-slave-5lgp"
server_key: "fecce8fc5754787cde6cf97c6c3f75f0"
server_key_iv: "51a7fe792ff7ce7a924d2812bb5519f3"
server_key_version: "encryptionkey@0"
I20260620 18:35:38.365262 24012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980527431848-22676-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980527431848-22676-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980527431848-22676-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:35:38.380959 24012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:35:38.384774 24012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:35:38.385051 24012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:35:38.387607 24012 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:35:38.387725 24012 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:38.387871 24012 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:35:38.387945 24012 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 18:35:38 dist-test-slave-5lgp krb5kdc[23756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980538, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.2@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM
I20260620 18:35:38.414705 23896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:38.388369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.2:55281 (local address 127.22.37.62:37885)
0620 18:35:38.388592 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:38.388600 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:38.390064 (+ 1464us) server_negotiation.cc:407] Connection header received
0620 18:35:38.392338 (+ 2274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:38.392345 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:38.392447 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:38.392576 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:38.395139 (+ 2563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:38.396252 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:38.399531 (+ 3279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:38.400204 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:38.400368 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:38.405921 (+ 5553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:38.405929 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:38.405933 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:38.406011 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:38.410099 (+ 4088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:38.411463 (+ 1364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:38.411471 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:38.411477 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:38.411646 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:38.412369 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:38.412374 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:38.412378 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:38.412847 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:38.413009 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:38.414266 (+ 1257us) server_negotiation.cc:299] Negotiation successful
0620 18:35:38.414440 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 18:35:38.416854 24036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:38.388963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:37885 (local address 127.22.37.2:55281)
0620 18:35:38.389890 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:38.389938 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:38.391958 (+ 2020us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:38.393017 (+ 1059us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:38.393031 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:38.393930 (+ 899us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:38.394926 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:38.394952 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:38.396440 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:38.396462 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:38.399322 (+ 2860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:38.399348 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:38.400484 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:38.400492 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:38.400744 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:38.401621 (+ 877us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:35:38.401667 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:38.405499 (+ 3832us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:35:38.410339 (+ 4840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:38.410354 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:38.410387 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:38.411279 (+ 892us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:38.411810 (+ 531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:38.411819 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:38.411825 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:38.412179 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:38.413096 (+ 917us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:35:38.413112 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:35:38.413758 (+ 646us) client_negotiation.cc:769] Sending connection context
0620 18:35:38.414150 (+ 392us) client_negotiation.cc:240] Negotiation successful
0620 18:35:38.414902 (+ 752us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":678,"thread_start_us":444,"threads_started":1}
I20260620 18:35:38.437189 23896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:38.423066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.2:44649 (local address 127.22.37.62:37885)
0620 18:35:38.423360 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:38.423369 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:38.423797 (+ 428us) server_negotiation.cc:407] Connection header received
0620 18:35:38.423994 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:38.424000 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:38.424075 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:38.424178 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:38.425078 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:38.425983 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:38.427705 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:38.428158 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:38.428294 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:38.430386 (+ 2092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:38.430391 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:38.430394 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:38.430444 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:38.433802 (+ 3358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:38.434771 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:38.434779 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:38.434784 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:38.434958 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:38.435459 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:38.435467 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:38.435471 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:38.435870 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:38.436067 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:38.436614 (+ 547us) server_negotiation.cc:299] Negotiation successful
0620 18:35:38.436847 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 18:35:38.437350 24114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:38.423015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:37885 (local address 127.22.37.2:44649)
0620 18:35:38.423690 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:38.423708 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:38.423872 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:38.424209 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:38.424213 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:38.424652 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:38.424937 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:38.424946 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:38.426178 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:38.426182 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:38.427566 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:38.427578 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:38.428357 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:38.428364 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:38.428523 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:38.429145 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:35:38.429162 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:38.430202 (+ 1040us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:35:38.433917 (+ 3715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:38.433926 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:38.433931 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:38.434646 (+ 715us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:38.435096 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:38.435104 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:38.435114 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:38.435329 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:38.436142 (+ 813us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:35:38.436147 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:35:38.436357 (+ 210us) client_negotiation.cc:769] Sending connection context
0620 18:35:38.436528 (+ 171us) client_negotiation.cc:240] Negotiation successful
0620 18:35:38.436871 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":355,"threads_started":1}
I20260620 18:35:38.444499 24012 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.2:37905
I20260620 18:35:38.444506 24146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.2:37905 every 8 connection(s)
I20260620 18:35:38.445825 24012 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980527431848-22676-0/minicluster-data/ts-1/data/info.pb
I20260620 18:35:38.447835 24147 heartbeater.cc:347] Connected to a master server at 127.22.37.62:37885
I20260620 18:35:38.448153 24147 heartbeater.cc:464] Registering TS with master...
I20260620 18:35:38.448819 24147 heartbeater.cc:511] Master 127.22.37.62:37885 requested a full tablet report, sending...
I20260620 18:35:38.450139 23801 ts_manager.cc:194] Registered new tserver with Master: 82ee0b3ce78a4e2c996a42b9644afd48 (127.22.37.2:37905)
I20260620 18:35:38.450918 23801 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.37.2@KRBTEST.COM'} at 127.22.37.2:55281
I20260620 18:35:38.452653 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 24012
I20260620 18:35:38.452865 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980527431848-22676-0/minicluster-data/ts-1/wal/instance
I20260620 18:35:38.453444 22676 external_mini_cluster.cc:1428] Setting key d4e6c2d67d7e5256f446d35646155fda
WARNING: no policy specified for kudu/127.22.37.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.37.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.37.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.3.keytab.
Entry for principal kudu/127.22.37.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.3.keytab.
I20260620 18:35:38.492329 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980527431848-22676-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980527431848-22676-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980527431848-22676-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980527431848-22676-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.37.3:0
--local_ip_for_outbound_sockets=127.22.37.3
--webserver_interface=127.22.37.3
--webserver_port=0
--tserver_master_addrs=127.22.37.62:37885
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.3.keytab
--principal=kudu/127.22.37.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:45519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:35:38.816968 24151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:35:38.817308 24151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:35:38.817487 24151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:35:38.824894 24151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:35:38.825053 24151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:35:38.825206 24151 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 18:35:38.825345 24151 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.37.3
I20260620 18:35:38.835531 24151 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:45519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980527431848-22676-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980527431848-22676-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.3.keytab
--principal=kudu/127.22.37.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.37.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980527431848-22676-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.37.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.37.62:37885
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.37.3
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980527431848-22676-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:35:38.836910 24151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:35:38.838306 24151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:35:38.851569 24156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:38.851492 24157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:38.851532 24159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:35:38.852689 24151 server_base.cc:1034] running on GCE node
I20260620 18:35:38.853293 24151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:35:38.854207 24151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:35:38.855445 24151 hybrid_clock.cc:648] HybridClock initialized: now 1781980538855392 us; error 47 us; skew 500 ppm
I20260620 18:35:38.855741 24151 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:35:38 dist-test-slave-5lgp krb5kdc[23756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980538, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:35:38.858834 24151 init.cc:377] Logged in from keytab as kudu/127.22.37.3@KRBTEST.COM (short username kudu)
I20260620 18:35:38.860276 24151 webserver.cc:460] Webserver started at http://127.22.37.3:36757/ using document root <none> and password file <none>
I20260620 18:35:38.860961 24151 fs_manager.cc:359] Metadata directory not provided
I20260620 18:35:38.861083 24151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:35:38.861447 24151 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:35:38.864239 24151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980527431848-22676-0/minicluster-data/ts-2/data/instance:
uuid: "c34aba1d518346b0937a687c4f9e9564"
format_stamp: "Formatted at 2026-06-20 18:35:38 on dist-test-slave-5lgp"
server_key: "af7c2fd07427d97156529d8bffb7cf4c"
server_key_iv: "16f3f8d62c5ef600051b3137f6e07562"
server_key_version: "encryptionkey@0"
I20260620 18:35:38.864992 24151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980527431848-22676-0/minicluster-data/ts-2/wal/instance:
uuid: "c34aba1d518346b0937a687c4f9e9564"
format_stamp: "Formatted at 2026-06-20 18:35:38 on dist-test-slave-5lgp"
server_key: "af7c2fd07427d97156529d8bffb7cf4c"
server_key_iv: "16f3f8d62c5ef600051b3137f6e07562"
server_key_version: "encryptionkey@0"
I20260620 18:35:38.870177 24151 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260620 18:35:38.873430 24166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:38.874675 24151 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:35:38.874847 24151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980527431848-22676-0/minicluster-data/ts-2/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980527431848-22676-0/minicluster-data/ts-2/wal
uuid: "c34aba1d518346b0937a687c4f9e9564"
format_stamp: "Formatted at 2026-06-20 18:35:38 on dist-test-slave-5lgp"
server_key: "af7c2fd07427d97156529d8bffb7cf4c"
server_key_iv: "16f3f8d62c5ef600051b3137f6e07562"
server_key_version: "encryptionkey@0"
I20260620 18:35:38.875054 24151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980527431848-22676-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980527431848-22676-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980527431848-22676-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:35:38.886339 24151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:35:38.890162 24151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:35:38.890432 24151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:35:38.893028 24151 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:35:38.893131 24151 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:38.893302 24151 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:35:38.893429 24151 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Jun 20 18:35:38 dist-test-slave-5lgp krb5kdc[23756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980538, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.3@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM
I20260620 18:35:38.918409 23896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:38.893477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.3:44141 (local address 127.22.37.62:37885)
0620 18:35:38.893676 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:38.893684 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:38.895063 (+ 1379us) server_negotiation.cc:407] Connection header received
0620 18:35:38.897011 (+ 1948us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:38.897018 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:38.897107 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:38.897212 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:38.899330 (+ 2118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:38.900626 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:38.903519 (+ 2893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:38.903937 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:38.904067 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:38.909560 (+ 5493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:38.909567 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:38.909574 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:38.909625 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:38.913686 (+ 4061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:38.914950 (+ 1264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:38.914958 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:38.914963 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:38.915136 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:38.915787 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:38.915795 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:38.915800 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:38.916217 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:38.916408 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:38.917615 (+ 1207us) server_negotiation.cc:299] Negotiation successful
0620 18:35:38.917958 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 18:35:38.920552 24175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:38.894148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:37885 (local address 127.22.37.3:44141)
0620 18:35:38.894902 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:38.894938 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:38.896719 (+ 1781us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:38.897494 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:38.897502 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:38.898282 (+ 780us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:38.899108 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:38.899152 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:38.900819 (+ 1667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:38.900830 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:38.903370 (+ 2540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:38.903384 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:38.904226 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:38.904234 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:38.904446 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:38.905192 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:35:38.905231 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:38.909241 (+ 4010us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:35:38.913853 (+ 4612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:38.913867 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:38.913888 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:38.914784 (+ 896us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:38.915262 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:38.915270 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:38.915275 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:38.915609 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:38.916400 (+ 791us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:35:38.916410 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:35:38.917054 (+ 644us) client_negotiation.cc:769] Sending connection context
0620 18:35:38.917490 (+ 436us) client_negotiation.cc:240] Negotiation successful
0620 18:35:38.918271 (+ 781us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":341,"threads_started":1}
I20260620 18:35:38.932767 24151 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.3:39949
I20260620 18:35:38.932868 24285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.3:39949 every 8 connection(s)
I20260620 18:35:38.934494 24151 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980527431848-22676-0/minicluster-data/ts-2/data/info.pb
I20260620 18:35:38.937237 24286 heartbeater.cc:347] Connected to a master server at 127.22.37.62:37885
I20260620 18:35:38.937624 24286 heartbeater.cc:464] Registering TS with master...
I20260620 18:35:38.938580 24286 heartbeater.cc:511] Master 127.22.37.62:37885 requested a full tablet report, sending...
I20260620 18:35:38.939788 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 24151
I20260620 18:35:38.939989 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980527431848-22676-0/minicluster-data/ts-2/wal/instance
I20260620 18:35:38.940013 23801 ts_manager.cc:194] Registered new tserver with Master: c34aba1d518346b0937a687c4f9e9564 (127.22.37.3:39949)
I20260620 18:35:38.940524 22676 external_mini_cluster.cc:1428] Setting key 855605fa5e0df35b7c78b7a1d59de566
I20260620 18:35:38.941118 23801 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.37.3@KRBTEST.COM'} at 127.22.37.3:44141
I20260620 18:35:38.942152 24271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:38.927306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:37885 (local address 127.22.37.3:41625)
0620 18:35:38.927955 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:38.927985 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:38.928150 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:38.928601 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:38.928608 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:38.929178 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:38.929761 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:38.929772 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:38.931123 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:38.931129 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:38.932648 (+ 1519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:38.932667 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:38.933627 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:38.933634 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:38.933827 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:38.934490 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:35:38.934508 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:38.935506 (+ 998us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:35:38.939516 (+ 4010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:38.939521 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:38.939525 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:38.939975 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:38.940371 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:38.940378 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:38.940381 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:38.940507 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:38.940989 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:35:38.940994 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:35:38.941187 (+ 193us) client_negotiation.cc:769] Sending connection context
0620 18:35:38.941332 (+ 145us) client_negotiation.cc:240] Negotiation successful
0620 18:35:38.941676 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":456,"threads_started":1}
I20260620 18:35:38.943152 23896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:38.927286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.3:41625 (local address 127.22.37.62:37885)
0620 18:35:38.927747 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:38.927756 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:38.928089 (+ 333us) server_negotiation.cc:407] Connection header received
0620 18:35:38.928307 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:38.928316 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:38.928433 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:38.928547 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:38.929917 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:38.930977 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:38.932865 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:38.933409 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:38.933549 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:38.935736 (+ 2187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:38.935783 (+ 47us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:38.935787 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:38.935835 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:38.939387 (+ 3552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:38.940096 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:38.940102 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:38.940106 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:38.940267 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:38.940609 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:38.940615 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:38.940618 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:38.940880 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:38.941025 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:38.942542 (+ 1517us) server_negotiation.cc:299] Negotiation successful
0620 18:35:38.942787 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304}
I20260620 18:35:38.943689 22676 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:35:38.946195 24008 heartbeater.cc:503] Master 127.22.37.62:37885 was elected leader, sending a full tablet report...
Jun 20 18:35:38 dist-test-slave-5lgp krb5kdc[23756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980538, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 18:35:38 dist-test-slave-5lgp krb5kdc[23756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980538, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM
I20260620 18:35:38.989045 23896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:38.975648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41556 (local address 127.22.37.62:37885)
0620 18:35:38.975838 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:38.975843 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:38.976099 (+ 256us) server_negotiation.cc:407] Connection header received
0620 18:35:38.976309 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:38.976313 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:38.976380 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:38.976472 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:38.977959 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:38.978792 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:38.980084 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:38.980510 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:38.980607 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:38.984281 (+ 3674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:38.984286 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:38.984289 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:38.984325 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:38.986899 (+ 2574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:38.987499 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:38.987503 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:38.987505 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:38.987626 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:38.987941 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:38.987944 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:38.987946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:38.988198 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:38.988374 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:38.988614 (+ 240us) server_negotiation.cc:299] Negotiation successful
0620 18:35:38.988776 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 18:35:38.992995 23800 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41556:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:35:38.999783 23800 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:35:39.031678 24300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:39.012306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52056 (local address 127.22.37.3:39949)
0620 18:35:39.019729 (+ 7423us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:39.019743 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:39.021203 (+ 1460us) server_negotiation.cc:407] Connection header received
0620 18:35:39.021330 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:39.021342 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:39.021675 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:39.021889 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:35:39.022922 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:39.025201 (+ 2279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:39.029081 (+ 3880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:39.030485 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:39.030628 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:39.030926 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:39.031179 (+ 253us) server_negotiation.cc:299] Negotiation successful
0620 18:35:39.031340 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7197,"thread_start_us":7015,"threads_started":1}
I20260620 18:35:39.031693 24301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:39.012010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.3:39949 (local address 127.0.0.1:52056)
0620 18:35:39.020840 (+ 8830us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:39.020904 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:39.021116 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:39.022084 (+ 968us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:39.022090 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:39.022119 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:35:39.022639 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:39.022656 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:39.025424 (+ 2768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:39.025434 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:39.028862 (+ 3428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:39.028874 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:39.030671 (+ 1797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:39.030677 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:39.030926 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:39.030992 (+ 66us) client_negotiation.cc:769] Sending connection context
0620 18:35:39.031132 (+ 140us) client_negotiation.cc:240] Negotiation successful
0620 18:35:39.031330 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8570,"thread_start_us":9458,"threads_started":1}
I20260620 18:35:39.036065 24303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:39.012752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.1:44765 (local address 127.0.0.1:56080)
0620 18:35:39.028110 (+ 15358us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:39.028131 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:39.028367 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:39.029327 (+ 960us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:39.029332 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:39.029386 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:35:39.029909 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:39.029922 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:39.031784 (+ 1862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:39.031791 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:39.033769 (+ 1978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:39.033783 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:39.035390 (+ 1607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:39.035395 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:39.035516 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:39.035538 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 18:35:39.035602 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 18:35:39.035698 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15219,"mutex_wait_us":99,"thread_start_us":14998,"threads_started":1}
I20260620 18:35:39.036209 24304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:39.013260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56080 (local address 127.22.37.1:44765)
0620 18:35:39.020211 (+ 6951us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:39.020224 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:39.028254 (+ 8030us) server_negotiation.cc:407] Connection header received
0620 18:35:39.028619 (+ 365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:39.028628 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:39.028872 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:39.029062 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:35:39.030081 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:39.031602 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:39.033966 (+ 2364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:39.035006 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:39.035089 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:39.035238 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:39.035711 (+ 473us) server_negotiation.cc:299] Negotiation successful
0620 18:35:39.035905 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6731,"thread_start_us":6510,"threads_started":1}
I20260620 18:35:39.035996 24216 tablet_service.cc:1467] Processing CreateTablet for tablet 2ba7c4c31a994f348d98ab6280ac4612 (DEFAULT_TABLE table=test-table [id=be60b868133e487bab9c87d9779daa33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:35:39.037657 24216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ba7c4c31a994f348d98ab6280ac4612. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:35:39.039806 24299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:39.012323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.2:37905 (local address 127.0.0.1:55758)
0620 18:35:39.021779 (+ 9456us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:39.021799 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:39.021916 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:39.029255 (+ 7339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:39.029260 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:39.029286 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:35:39.029768 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:39.029781 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:39.031699 (+ 1918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:39.031706 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:39.036970 (+ 5264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:39.036982 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:39.038994 (+ 2012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:39.039001 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:39.039173 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:39.039207 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 18:35:39.039300 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 18:35:39.039454 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9299,"mutex_wait_us":77,"thread_start_us":8128,"threads_started":1}
I20260620 18:35:39.039901 24302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:39.012614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55758 (local address 127.22.37.2:37905)
0620 18:35:39.028406 (+ 15792us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:39.028417 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:39.028444 (+ 27us) server_negotiation.cc:407] Connection header received
0620 18:35:39.028554 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:39.028568 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:39.028813 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:39.029048 (+ 235us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:35:39.030018 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:39.031534 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:39.037132 (+ 5598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:39.038534 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:39.038665 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:39.038853 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:39.039454 (+ 601us) server_negotiation.cc:299] Negotiation successful
0620 18:35:39.039641 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15633,"thread_start_us":15493,"threads_started":1}
I20260620 18:35:39.040355 23938 tablet_service.cc:1467] Processing CreateTablet for tablet 2ba7c4c31a994f348d98ab6280ac4612 (DEFAULT_TABLE table=test-table [id=be60b868133e487bab9c87d9779daa33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:35:39.042371 23938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ba7c4c31a994f348d98ab6280ac4612. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:35:39.043810 24077 tablet_service.cc:1467] Processing CreateTablet for tablet 2ba7c4c31a994f348d98ab6280ac4612 (DEFAULT_TABLE table=test-table [id=be60b868133e487bab9c87d9779daa33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:35:39.045167 24077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ba7c4c31a994f348d98ab6280ac4612. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:35:39.050093 24305 tablet_bootstrap.cc:492] T 2ba7c4c31a994f348d98ab6280ac4612 P c34aba1d518346b0937a687c4f9e9564: Bootstrap starting.
I20260620 18:35:39.053357 24305 tablet_bootstrap.cc:654] T 2ba7c4c31a994f348d98ab6280ac4612 P c34aba1d518346b0937a687c4f9e9564: Neither blocks nor log segments found. Creating new log.
I20260620 18:35:39.054450 24305 log.cc:824] T 2ba7c4c31a994f348d98ab6280ac4612 P c34aba1d518346b0937a687c4f9e9564: Log is configured to *not* fsync() on all Append() calls
I20260620 18:35:39.058048 24307 tablet_bootstrap.cc:492] T 2ba7c4c31a994f348d98ab6280ac4612 P 2fdb54ee16854fcbbb83f8a380d57e5d: Bootstrap starting.
I20260620 18:35:39.061062 24308 tablet_bootstrap.cc:492] T 2ba7c4c31a994f348d98ab6280ac4612 P 82ee0b3ce78a4e2c996a42b9644afd48: Bootstrap starting.
I20260620 18:35:39.061106 24305 tablet_bootstrap.cc:492] T 2ba7c4c31a994f348d98ab6280ac4612 P c34aba1d518346b0937a687c4f9e9564: No bootstrap required, opened a new log
I20260620 18:35:39.061352 24305 ts_tablet_manager.cc:1397] T 2ba7c4c31a994f348d98ab6280ac4612 P c34aba1d518346b0937a687c4f9e9564: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.005s
I20260620 18:35:39.061736 24307 tablet_bootstrap.cc:654] T 2ba7c4c31a994f348d98ab6280ac4612 P 2fdb54ee16854fcbbb83f8a380d57e5d: Neither blocks nor log segments found. Creating new log.
I20260620 18:35:39.062961 24307 log.cc:824] T 2ba7c4c31a994f348d98ab6280ac4612 P 2fdb54ee16854fcbbb83f8a380d57e5d: Log is configured to *not* fsync() on all Append() calls
I20260620 18:35:39.063951 24308 tablet_bootstrap.cc:654] T 2ba7c4c31a994f348d98ab6280ac4612 P 82ee0b3ce78a4e2c996a42b9644afd48: Neither blocks nor log segments found. Creating new log.
I20260620 18:35:39.064973 24308 log.cc:824] T 2ba7c4c31a994f348d98ab6280ac4612 P 82ee0b3ce78a4e2c996a42b9644afd48: Log is configured to *not* fsync() on all Append() calls
I20260620 18:35:39.067597 24307 tablet_bootstrap.cc:492] T 2ba7c4c31a994f348d98ab6280ac4612 P 2fdb54ee16854fcbbb83f8a380d57e5d: No bootstrap required, opened a new log
I20260620 18:35:39.067849 24308 tablet_bootstrap.cc:492] T 2ba7c4c31a994f348d98ab6280ac4612 P 82ee0b3ce78a4e2c996a42b9644afd48: No bootstrap required, opened a new log
I20260620 18:35:39.067864 24307 ts_tablet_manager.cc:1397] T 2ba7c4c31a994f348d98ab6280ac4612 P 2fdb54ee16854fcbbb83f8a380d57e5d: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.005s
I20260620 18:35:39.068110 24308 ts_tablet_manager.cc:1397] T 2ba7c4c31a994f348d98ab6280ac4612 P 82ee0b3ce78a4e2c996a42b9644afd48: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20260620 18:35:39.072597 24305 raft_consensus.cc:348] T 2ba7c4c31a994f348d98ab6280ac4612 P c34aba1d518346b0937a687c4f9e9564 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c34aba1d518346b0937a687c4f9e9564" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 39949 } } peers { permanent_uuid: "82ee0b3ce78a4e2c996a42b9644afd48" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 37905 } } peers { permanent_uuid: "2fdb54ee16854fcbbb83f8a380d57e5d" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44765 } }
I20260620 18:35:39.073060 24305 raft_consensus.cc:374] T 2ba7c4c31a994f348d98ab6280ac4612 P c34aba1d518346b0937a687c4f9e9564 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:35:39.073163 24305 raft_consensus.cc:729] T 2ba7c4c31a994f348d98ab6280ac4612 P c34aba1d518346b0937a687c4f9e9564 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c34aba1d518346b0937a687c4f9e9564, State: Initialized, Role: FOLLOWER
I20260620 18:35:39.073843 24305 consensus_queue.cc:260] T 2ba7c4c31a994f348d98ab6280ac4612 P c34aba1d518346b0937a687c4f9e9564 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c34aba1d518346b0937a687c4f9e9564" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 39949 } } peers { permanent_uuid: "82ee0b3ce78a4e2c996a42b9644afd48" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 37905 } } peers { permanent_uuid: "2fdb54ee16854fcbbb83f8a380d57e5d" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44765 } }
I20260620 18:35:39.075912 24307 raft_consensus.cc:348] T 2ba7c4c31a994f348d98ab6280ac4612 P 2fdb54ee16854fcbbb83f8a380d57e5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c34aba1d518346b0937a687c4f9e9564" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 39949 } } peers { permanent_uuid: "82ee0b3ce78a4e2c996a42b9644afd48" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 37905 } } peers { permanent_uuid: "2fdb54ee16854fcbbb83f8a380d57e5d" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44765 } }
I20260620 18:35:39.076251 24307 raft_consensus.cc:374] T 2ba7c4c31a994f348d98ab6280ac4612 P 2fdb54ee16854fcbbb83f8a380d57e5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:35:39.076359 24307 raft_consensus.cc:729] T 2ba7c4c31a994f348d98ab6280ac4612 P 2fdb54ee16854fcbbb83f8a380d57e5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fdb54ee16854fcbbb83f8a380d57e5d, State: Initialized, Role: FOLLOWER
I20260620 18:35:39.076924 24307 consensus_queue.cc:260] T 2ba7c4c31a994f348d98ab6280ac4612 P 2fdb54ee16854fcbbb83f8a380d57e5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c34aba1d518346b0937a687c4f9e9564" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 39949 } } peers { permanent_uuid: "82ee0b3ce78a4e2c996a42b9644afd48" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 37905 } } peers { permanent_uuid: "2fdb54ee16854fcbbb83f8a380d57e5d" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44765 } }
I20260620 18:35:39.078703 24307 ts_tablet_manager.cc:1428] T 2ba7c4c31a994f348d98ab6280ac4612 P 2fdb54ee16854fcbbb83f8a380d57e5d: Time spent starting tablet: real 0.011s user 0.011s sys 0.001s
I20260620 18:35:39.079799 24308 raft_consensus.cc:348] T 2ba7c4c31a994f348d98ab6280ac4612 P 82ee0b3ce78a4e2c996a42b9644afd48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c34aba1d518346b0937a687c4f9e9564" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 39949 } } peers { permanent_uuid: "82ee0b3ce78a4e2c996a42b9644afd48" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 37905 } } peers { permanent_uuid: "2fdb54ee16854fcbbb83f8a380d57e5d" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44765 } }
I20260620 18:35:39.080351 24308 raft_consensus.cc:374] T 2ba7c4c31a994f348d98ab6280ac4612 P 82ee0b3ce78a4e2c996a42b9644afd48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:35:39.080464 24308 raft_consensus.cc:729] T 2ba7c4c31a994f348d98ab6280ac4612 P 82ee0b3ce78a4e2c996a42b9644afd48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82ee0b3ce78a4e2c996a42b9644afd48, State: Initialized, Role: FOLLOWER
I20260620 18:35:39.081058 24308 consensus_queue.cc:260] T 2ba7c4c31a994f348d98ab6280ac4612 P 82ee0b3ce78a4e2c996a42b9644afd48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c34aba1d518346b0937a687c4f9e9564" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 39949 } } peers { permanent_uuid: "82ee0b3ce78a4e2c996a42b9644afd48" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 37905 } } peers { permanent_uuid: "2fdb54ee16854fcbbb83f8a380d57e5d" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44765 } }
I20260620 18:35:39.081211 24286 heartbeater.cc:503] Master 127.22.37.62:37885 was elected leader, sending a full tablet report...
I20260620 18:35:39.082304 24305 ts_tablet_manager.cc:1428] T 2ba7c4c31a994f348d98ab6280ac4612 P c34aba1d518346b0937a687c4f9e9564: Time spent starting tablet: real 0.021s user 0.012s sys 0.007s
I20260620 18:35:39.088023 24147 heartbeater.cc:503] Master 127.22.37.62:37885 was elected leader, sending a full tablet report...
I20260620 18:35:39.088940 24308 ts_tablet_manager.cc:1428] T 2ba7c4c31a994f348d98ab6280ac4612 P 82ee0b3ce78a4e2c996a42b9644afd48: Time spent starting tablet: real 0.021s user 0.016s sys 0.005s
W20260620 18:35:39.187952 24009 tablet.cc:2369] T 2ba7c4c31a994f348d98ab6280ac4612 P 2fdb54ee16854fcbbb83f8a380d57e5d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:35:39.188396 24287 tablet.cc:2369] T 2ba7c4c31a994f348d98ab6280ac4612 P c34aba1d518346b0937a687c4f9e9564: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:35:39.195463 24312 raft_consensus.cc:482] T 2ba7c4c31a994f348d98ab6280ac4612 P 2fdb54ee16854fcbbb83f8a380d57e5d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:35:39.195791 24312 raft_consensus.cc:504] T 2ba7c4c31a994f348d98ab6280ac4612 P 2fdb54ee16854fcbbb83f8a380d57e5d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c34aba1d518346b0937a687c4f9e9564" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 39949 } } peers { permanent_uuid: "82ee0b3ce78a4e2c996a42b9644afd48" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 37905 } } peers { permanent_uuid: "2fdb54ee16854fcbbb83f8a380d57e5d" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44765 } }
I20260620 18:35:39.197557 24312 leader_election.cc:288] T 2ba7c4c31a994f348d98ab6280ac4612 P 2fdb54ee16854fcbbb83f8a380d57e5d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c34aba1d518346b0937a687c4f9e9564 (127.22.37.3:39949), 82ee0b3ce78a4e2c996a42b9644afd48 (127.22.37.2:37905)
W20260620 18:35:39.197716 24148 tablet.cc:2369] T 2ba7c4c31a994f348d98ab6280ac4612 P 82ee0b3ce78a4e2c996a42b9644afd48: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:35:39.204463 24300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:39.198161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.1:37033 (local address 127.22.37.3:39949)
0620 18:35:39.198596 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:39.198602 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:39.198853 (+ 251us) server_negotiation.cc:407] Connection header received
0620 18:35:39.199127 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:39.199136 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:39.199222 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:39.199288 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:35:39.200149 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:39.200814 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:39.202717 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:39.203496 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:39.203653 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:39.203767 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:39.204045 (+ 278us) server_negotiation.cc:299] Negotiation successful
0620 18:35:39.204178 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271}
I20260620 18:35:39.204517 24302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:39.198840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.1:51169 (local address 127.22.37.2:37905)
0620 18:35:39.199051 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:39.199059 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:39.199116 (+ 57us) server_negotiation.cc:407] Connection header received
0620 18:35:39.199297 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:39.199302 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:39.199371 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:39.199465 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:35:39.200017 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:39.200792 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:39.202718 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:39.203459 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:39.203615 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:39.203725 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:39.204049 (+ 324us) server_negotiation.cc:299] Negotiation successful
0620 18:35:39.204180 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 18:35:39.204661 24314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:39.197888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.3:39949 (local address 127.22.37.1:37033)
0620 18:35:39.198734 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:39.198750 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:39.198951 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:39.199491 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:39.199495 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:39.199519 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:35:39.199987 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:39.200000 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:39.201011 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:39.201015 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:39.202577 (+ 1562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:39.202588 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:39.203728 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:39.203734 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:39.203894 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:39.203923 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 18:35:39.204020 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 18:35:39.204190 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":692,"mutex_wait_us":76,"thread_start_us":507,"threads_started":1}
I20260620 18:35:39.204993 24315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:39.198053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.2:37905 (local address 127.22.37.1:51169)
0620 18:35:39.199040 (+ 987us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:39.199056 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:39.199172 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:39.199478 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:39.199481 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:39.199498 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:35:39.199855 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:39.199865 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:39.200924 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:39.200932 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:39.202573 (+ 1641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:39.202585 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:39.203684 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:39.203690 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:39.203855 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:39.203919 (+ 64us) client_negotiation.cc:769] Sending connection context
0620 18:35:39.204022 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 18:35:39.204171 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":863,"mutex_wait_us":38,"thread_start_us":852,"threads_started":1}
I20260620 18:35:39.205869 24236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ba7c4c31a994f348d98ab6280ac4612" candidate_uuid: "2fdb54ee16854fcbbb83f8a380d57e5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c34aba1d518346b0937a687c4f9e9564" is_pre_election: true
I20260620 18:35:39.205960 24097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ba7c4c31a994f348d98ab6280ac4612" candidate_uuid: "2fdb54ee16854fcbbb83f8a380d57e5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82ee0b3ce78a4e2c996a42b9644afd48" is_pre_election: true
I20260620 18:35:39.206482 24097 raft_consensus.cc:2454] T 2ba7c4c31a994f348d98ab6280ac4612 P 82ee0b3ce78a4e2c996a42b9644afd48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2fdb54ee16854fcbbb83f8a380d57e5d in term 0.
I20260620 18:35:39.206483 24236 raft_consensus.cc:2454] T 2ba7c4c31a994f348d98ab6280ac4612 P c34aba1d518346b0937a687c4f9e9564 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2fdb54ee16854fcbbb83f8a380d57e5d in term 0.
I20260620 18:35:39.207214 23891 leader_election.cc:302] T 2ba7c4c31a994f348d98ab6280ac4612 P 2fdb54ee16854fcbbb83f8a380d57e5d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fdb54ee16854fcbbb83f8a380d57e5d, 82ee0b3ce78a4e2c996a42b9644afd48; no voters:
I20260620 18:35:39.207669 24312 raft_consensus.cc:2781] T 2ba7c4c31a994f348d98ab6280ac4612 P 2fdb54ee16854fcbbb83f8a380d57e5d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:35:39.207846 24312 raft_consensus.cc:482] T 2ba7c4c31a994f348d98ab6280ac4612 P 2fdb54ee16854fcbbb83f8a380d57e5d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:35:39.207971 24312 raft_consensus.cc:3037] T 2ba7c4c31a994f348d98ab6280ac4612 P 2fdb54ee16854fcbbb83f8a380d57e5d [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:35:39.210354 24312 raft_consensus.cc:504] T 2ba7c4c31a994f348d98ab6280ac4612 P 2fdb54ee16854fcbbb83f8a380d57e5d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c34aba1d518346b0937a687c4f9e9564" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 39949 } } peers { permanent_uuid: "82ee0b3ce78a4e2c996a42b9644afd48" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 37905 } } peers { permanent_uuid: "2fdb54ee16854fcbbb83f8a380d57e5d" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44765 } }
I20260620 18:35:39.211045 24312 leader_election.cc:288] T 2ba7c4c31a994f348d98ab6280ac4612 P 2fdb54ee16854fcbbb83f8a380d57e5d [CANDIDATE]: Term 1 election: Requested vote from peers c34aba1d518346b0937a687c4f9e9564 (127.22.37.3:39949), 82ee0b3ce78a4e2c996a42b9644afd48 (127.22.37.2:37905)
I20260620 18:35:39.211578 24236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ba7c4c31a994f348d98ab6280ac4612" candidate_uuid: "2fdb54ee16854fcbbb83f8a380d57e5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c34aba1d518346b0937a687c4f9e9564"
I20260620 18:35:39.211583 24097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ba7c4c31a994f348d98ab6280ac4612" candidate_uuid: "2fdb54ee16854fcbbb83f8a380d57e5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82ee0b3ce78a4e2c996a42b9644afd48"
I20260620 18:35:39.211825 24097 raft_consensus.cc:3037] T 2ba7c4c31a994f348d98ab6280ac4612 P 82ee0b3ce78a4e2c996a42b9644afd48 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:35:39.211856 24236 raft_consensus.cc:3037] T 2ba7c4c31a994f348d98ab6280ac4612 P c34aba1d518346b0937a687c4f9e9564 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:35:39.215267 24097 raft_consensus.cc:2454] T 2ba7c4c31a994f348d98ab6280ac4612 P 82ee0b3ce78a4e2c996a42b9644afd48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fdb54ee16854fcbbb83f8a380d57e5d in term 1.
I20260620 18:35:39.215667 24236 raft_consensus.cc:2454] T 2ba7c4c31a994f348d98ab6280ac4612 P c34aba1d518346b0937a687c4f9e9564 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fdb54ee16854fcbbb83f8a380d57e5d in term 1.
I20260620 18:35:39.215741 23891 leader_election.cc:302] T 2ba7c4c31a994f348d98ab6280ac4612 P 2fdb54ee16854fcbbb83f8a380d57e5d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fdb54ee16854fcbbb83f8a380d57e5d, 82ee0b3ce78a4e2c996a42b9644afd48; no voters:
I20260620 18:35:39.216058 24312 raft_consensus.cc:2781] T 2ba7c4c31a994f348d98ab6280ac4612 P 2fdb54ee16854fcbbb83f8a380d57e5d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:35:39.216522 24312 raft_consensus.cc:686] T 2ba7c4c31a994f348d98ab6280ac4612 P 2fdb54ee16854fcbbb83f8a380d57e5d [term 1 LEADER]: Becoming Leader. State: Replica: 2fdb54ee16854fcbbb83f8a380d57e5d, State: Running, Role: LEADER
I20260620 18:35:39.217056 24312 consensus_queue.cc:237] T 2ba7c4c31a994f348d98ab6280ac4612 P 2fdb54ee16854fcbbb83f8a380d57e5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c34aba1d518346b0937a687c4f9e9564" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 39949 } } peers { permanent_uuid: "82ee0b3ce78a4e2c996a42b9644afd48" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 37905 } } peers { permanent_uuid: "2fdb54ee16854fcbbb83f8a380d57e5d" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44765 } }
I20260620 18:35:39.221532 23801 catalog_manager.cc:5495] T 2ba7c4c31a994f348d98ab6280ac4612 P 2fdb54ee16854fcbbb83f8a380d57e5d reported cstate change: term changed from 0 to 1, leader changed from <none> to 2fdb54ee16854fcbbb83f8a380d57e5d (127.22.37.1). New cstate: current_term: 1 leader_uuid: "2fdb54ee16854fcbbb83f8a380d57e5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c34aba1d518346b0937a687c4f9e9564" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 39949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82ee0b3ce78a4e2c996a42b9644afd48" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 37905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2fdb54ee16854fcbbb83f8a380d57e5d" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44765 } health_report { overall_health: HEALTHY } } }
I20260620 18:35:39.278448 24304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:39.272785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56094 (local address 127.22.37.1:44765)
0620 18:35:39.273045 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:39.273052 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:39.273102 (+ 50us) server_negotiation.cc:407] Connection header received
0620 18:35:39.273425 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:39.273431 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:39.273519 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:39.273641 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:35:39.273986 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:39.274701 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:39.276076 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:39.276465 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:39.276562 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:39.277095 (+ 533us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:35:39.277776 (+ 681us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:35:39.277893 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:39.278049 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 18:35:39.278116 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 18:35:39.287026 24097 raft_consensus.cc:1261] T 2ba7c4c31a994f348d98ab6280ac4612 P 82ee0b3ce78a4e2c996a42b9644afd48 [term 1 FOLLOWER]: Refusing update from remote peer 2fdb54ee16854fcbbb83f8a380d57e5d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:35:39.287040 24236 raft_consensus.cc:1261] T 2ba7c4c31a994f348d98ab6280ac4612 P c34aba1d518346b0937a687c4f9e9564 [term 1 FOLLOWER]: Refusing update from remote peer 2fdb54ee16854fcbbb83f8a380d57e5d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:35:39.288000 24316 consensus_queue.cc:1035] T 2ba7c4c31a994f348d98ab6280ac4612 P 2fdb54ee16854fcbbb83f8a380d57e5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "82ee0b3ce78a4e2c996a42b9644afd48" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 37905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:35:39.288365 24312 consensus_queue.cc:1035] T 2ba7c4c31a994f348d98ab6280ac4612 P 2fdb54ee16854fcbbb83f8a380d57e5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c34aba1d518346b0937a687c4f9e9564" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:35:39.298633 24320 mvcc.cc:205] Tried to move back new op lower bound from 7298992288909602816 to 7298992288636440576. Current Snapshot: MvccSnapshot[applied={T|T < 7298992288909602816}]
I20260620 18:35:39.302589 24319 mvcc.cc:205] Tried to move back new op lower bound from 7298992288909602816 to 7298992288636440576. Current Snapshot: MvccSnapshot[applied={T|T < 7298992288909602816}]
I20260620 18:35:39.312986 24321 mvcc.cc:205] Tried to move back new op lower bound from 7298992288909602816 to 7298992288636440576. Current Snapshot: MvccSnapshot[applied={T|T < 7298992288909602816}]
I20260620 18:35:39.328590 23801 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41556:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:35:39.328855 23801 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41556:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:35:39.332201 23801 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f72af03185aa4be6bfacf552ced6b7a4: Sending DeleteTablet for 3 replicas of tablet 2ba7c4c31a994f348d98ab6280ac4612
I20260620 18:35:39.333395 24216 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ba7c4c31a994f348d98ab6280ac4612 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:35:39 UTC) from {username='kudu', principal='kudu/127.22.37.62@KRBTEST.COM'} at 127.0.0.1:52056
I20260620 18:35:39.333662 23938 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ba7c4c31a994f348d98ab6280ac4612 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:35:39 UTC) from {username='kudu', principal='kudu/127.22.37.62@KRBTEST.COM'} at 127.0.0.1:56080
I20260620 18:35:39.334134 24077 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ba7c4c31a994f348d98ab6280ac4612 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:35:39 UTC) from {username='kudu', principal='kudu/127.22.37.62@KRBTEST.COM'} at 127.0.0.1:55758
I20260620 18:35:39.334649 24329 tablet_replica.cc:330] T 2ba7c4c31a994f348d98ab6280ac4612 P 2fdb54ee16854fcbbb83f8a380d57e5d: stopping tablet replica
I20260620 18:35:39.335147 24329 raft_consensus.cc:2229] T 2ba7c4c31a994f348d98ab6280ac4612 P 2fdb54ee16854fcbbb83f8a380d57e5d [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:35:39.336325 24329 raft_consensus.cc:2258] T 2ba7c4c31a994f348d98ab6280ac4612 P 2fdb54ee16854fcbbb83f8a380d57e5d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:35:39.336969 23801 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41556:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:35:39.337688 23801 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:35:39.339159 24329 ts_tablet_manager.cc:1905] T 2ba7c4c31a994f348d98ab6280ac4612 P 2fdb54ee16854fcbbb83f8a380d57e5d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:35:39.340585 24328 tablet_replica.cc:330] T 2ba7c4c31a994f348d98ab6280ac4612 P c34aba1d518346b0937a687c4f9e9564: stopping tablet replica
I20260620 18:35:39.341032 24328 raft_consensus.cc:2229] T 2ba7c4c31a994f348d98ab6280ac4612 P c34aba1d518346b0937a687c4f9e9564 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:35:39.342818 24328 raft_consensus.cc:2258] T 2ba7c4c31a994f348d98ab6280ac4612 P c34aba1d518346b0937a687c4f9e9564 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:35:39.347064 24328 ts_tablet_manager.cc:1905] T 2ba7c4c31a994f348d98ab6280ac4612 P c34aba1d518346b0937a687c4f9e9564: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:35:39.347050 24076 tablet_service.cc:1467] Processing CreateTablet for tablet ede9baf238e24b538e3d0a2459a8826f (DEFAULT_TABLE table=test-table [id=a7a9cc630f7b428bb49b5fd6bc40fb83]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:35:39.347627 24076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ede9baf238e24b538e3d0a2459a8826f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:35:39.348513 24329 ts_tablet_manager.cc:1918] T 2ba7c4c31a994f348d98ab6280ac4612 P 2fdb54ee16854fcbbb83f8a380d57e5d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:35:39.348767 24329 log.cc:1196] T 2ba7c4c31a994f348d98ab6280ac4612 P 2fdb54ee16854fcbbb83f8a380d57e5d: Deleting WAL directory at /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980527431848-22676-0/minicluster-data/ts-0/wal/wals/2ba7c4c31a994f348d98ab6280ac4612
I20260620 18:35:39.349421 24329 ts_tablet_manager.cc:1939] T 2ba7c4c31a994f348d98ab6280ac4612 P 2fdb54ee16854fcbbb83f8a380d57e5d: Deleting consensus metadata
I20260620 18:35:39.353693 23787 catalog_manager.cc:4841] TS 2fdb54ee16854fcbbb83f8a380d57e5d (127.22.37.1:44765): tablet 2ba7c4c31a994f348d98ab6280ac4612 (table test-table [id=be60b868133e487bab9c87d9779daa33]) successfully deleted
I20260620 18:35:39.354054 23938 tablet_service.cc:1467] Processing CreateTablet for tablet ede9baf238e24b538e3d0a2459a8826f (DEFAULT_TABLE table=test-table [id=a7a9cc630f7b428bb49b5fd6bc40fb83]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:35:39.354516 23938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ede9baf238e24b538e3d0a2459a8826f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:35:39.355603 24328 ts_tablet_manager.cc:1918] T 2ba7c4c31a994f348d98ab6280ac4612 P c34aba1d518346b0937a687c4f9e9564: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:35:39.355758 24328 log.cc:1196] T 2ba7c4c31a994f348d98ab6280ac4612 P c34aba1d518346b0937a687c4f9e9564: Deleting WAL directory at /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980527431848-22676-0/minicluster-data/ts-2/wal/wals/2ba7c4c31a994f348d98ab6280ac4612
I20260620 18:35:39.356267 24328 ts_tablet_manager.cc:1939] T 2ba7c4c31a994f348d98ab6280ac4612 P c34aba1d518346b0937a687c4f9e9564: Deleting consensus metadata
I20260620 18:35:39.360579 24330 tablet_replica.cc:330] T 2ba7c4c31a994f348d98ab6280ac4612 P 82ee0b3ce78a4e2c996a42b9644afd48: stopping tablet replica
I20260620 18:35:39.361039 24330 raft_consensus.cc:2229] T 2ba7c4c31a994f348d98ab6280ac4612 P 82ee0b3ce78a4e2c996a42b9644afd48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:35:39.361786 24330 raft_consensus.cc:2258] T 2ba7c4c31a994f348d98ab6280ac4612 P 82ee0b3ce78a4e2c996a42b9644afd48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:35:39.362619 24216 tablet_service.cc:1467] Processing CreateTablet for tablet ede9baf238e24b538e3d0a2459a8826f (DEFAULT_TABLE table=test-table [id=a7a9cc630f7b428bb49b5fd6bc40fb83]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:35:39.363063 24216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ede9baf238e24b538e3d0a2459a8826f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:35:39.363965 24330 ts_tablet_manager.cc:1905] T 2ba7c4c31a994f348d98ab6280ac4612 P 82ee0b3ce78a4e2c996a42b9644afd48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:35:39.364789 24307 tablet_bootstrap.cc:492] T ede9baf238e24b538e3d0a2459a8826f P 2fdb54ee16854fcbbb83f8a380d57e5d: Bootstrap starting.
I20260620 18:35:39.365592 24308 tablet_bootstrap.cc:492] T ede9baf238e24b538e3d0a2459a8826f P 82ee0b3ce78a4e2c996a42b9644afd48: Bootstrap starting.
I20260620 18:35:39.366472 24307 tablet_bootstrap.cc:654] T ede9baf238e24b538e3d0a2459a8826f P 2fdb54ee16854fcbbb83f8a380d57e5d: Neither blocks nor log segments found. Creating new log.
I20260620 18:35:39.366900 23785 catalog_manager.cc:4841] TS c34aba1d518346b0937a687c4f9e9564 (127.22.37.3:39949): tablet 2ba7c4c31a994f348d98ab6280ac4612 (table test-table [id=be60b868133e487bab9c87d9779daa33]) successfully deleted
I20260620 18:35:39.367209 24308 tablet_bootstrap.cc:654] T ede9baf238e24b538e3d0a2459a8826f P 82ee0b3ce78a4e2c996a42b9644afd48: Neither blocks nor log segments found. Creating new log.
I20260620 18:35:39.373039 24305 tablet_bootstrap.cc:492] T ede9baf238e24b538e3d0a2459a8826f P c34aba1d518346b0937a687c4f9e9564: Bootstrap starting.
I20260620 18:35:39.374851 24305 tablet_bootstrap.cc:654] T ede9baf238e24b538e3d0a2459a8826f P c34aba1d518346b0937a687c4f9e9564: Neither blocks nor log segments found. Creating new log.
I20260620 18:35:39.378088 24330 ts_tablet_manager.cc:1918] T 2ba7c4c31a994f348d98ab6280ac4612 P 82ee0b3ce78a4e2c996a42b9644afd48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:35:39.378293 24330 log.cc:1196] T 2ba7c4c31a994f348d98ab6280ac4612 P 82ee0b3ce78a4e2c996a42b9644afd48: Deleting WAL directory at /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980527431848-22676-0/minicluster-data/ts-1/wal/wals/2ba7c4c31a994f348d98ab6280ac4612
I20260620 18:35:39.378952 24330 ts_tablet_manager.cc:1939] T 2ba7c4c31a994f348d98ab6280ac4612 P 82ee0b3ce78a4e2c996a42b9644afd48: Deleting consensus metadata
I20260620 18:35:39.381050 23788 catalog_manager.cc:4841] TS 82ee0b3ce78a4e2c996a42b9644afd48 (127.22.37.2:37905): tablet 2ba7c4c31a994f348d98ab6280ac4612 (table test-table [id=be60b868133e487bab9c87d9779daa33]) successfully deleted
I20260620 18:35:39.382654 24308 tablet_bootstrap.cc:492] T ede9baf238e24b538e3d0a2459a8826f P 82ee0b3ce78a4e2c996a42b9644afd48: No bootstrap required, opened a new log
I20260620 18:35:39.382831 24308 ts_tablet_manager.cc:1397] T ede9baf238e24b538e3d0a2459a8826f P 82ee0b3ce78a4e2c996a42b9644afd48: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.004s
I20260620 18:35:39.383860 24308 raft_consensus.cc:348] T ede9baf238e24b538e3d0a2459a8826f P 82ee0b3ce78a4e2c996a42b9644afd48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fdb54ee16854fcbbb83f8a380d57e5d" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44765 } } peers { permanent_uuid: "82ee0b3ce78a4e2c996a42b9644afd48" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 37905 } } peers { permanent_uuid: "c34aba1d518346b0937a687c4f9e9564" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 39949 } }
I20260620 18:35:39.384166 24308 raft_consensus.cc:374] T ede9baf238e24b538e3d0a2459a8826f P 82ee0b3ce78a4e2c996a42b9644afd48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:35:39.384280 24308 raft_consensus.cc:729] T ede9baf238e24b538e3d0a2459a8826f P 82ee0b3ce78a4e2c996a42b9644afd48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82ee0b3ce78a4e2c996a42b9644afd48, State: Initialized, Role: FOLLOWER
I20260620 18:35:39.384639 24308 consensus_queue.cc:260] T ede9baf238e24b538e3d0a2459a8826f P 82ee0b3ce78a4e2c996a42b9644afd48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fdb54ee16854fcbbb83f8a380d57e5d" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44765 } } peers { permanent_uuid: "82ee0b3ce78a4e2c996a42b9644afd48" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 37905 } } peers { permanent_uuid: "c34aba1d518346b0937a687c4f9e9564" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 39949 } }
I20260620 18:35:39.385402 24308 ts_tablet_manager.cc:1428] T ede9baf238e24b538e3d0a2459a8826f P 82ee0b3ce78a4e2c996a42b9644afd48: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260620 18:35:39.387008 24307 tablet_bootstrap.cc:492] T ede9baf238e24b538e3d0a2459a8826f P 2fdb54ee16854fcbbb83f8a380d57e5d: No bootstrap required, opened a new log
I20260620 18:35:39.387189 24307 ts_tablet_manager.cc:1397] T ede9baf238e24b538e3d0a2459a8826f P 2fdb54ee16854fcbbb83f8a380d57e5d: Time spent bootstrapping tablet: real 0.022s user 0.001s sys 0.019s
I20260620 18:35:39.388092 24307 raft_consensus.cc:348] T ede9baf238e24b538e3d0a2459a8826f P 2fdb54ee16854fcbbb83f8a380d57e5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fdb54ee16854fcbbb83f8a380d57e5d" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44765 } } peers { permanent_uuid: "82ee0b3ce78a4e2c996a42b9644afd48" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 37905 } } peers { permanent_uuid: "c34aba1d518346b0937a687c4f9e9564" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 39949 } }
I20260620 18:35:39.388327 24307 raft_consensus.cc:374] T ede9baf238e24b538e3d0a2459a8826f P 2fdb54ee16854fcbbb83f8a380d57e5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:35:39.388458 24305 tablet_bootstrap.cc:492] T ede9baf238e24b538e3d0a2459a8826f P c34aba1d518346b0937a687c4f9e9564: No bootstrap required, opened a new log
I20260620 18:35:39.388443 24307 raft_consensus.cc:729] T ede9baf238e24b538e3d0a2459a8826f P 2fdb54ee16854fcbbb83f8a380d57e5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fdb54ee16854fcbbb83f8a380d57e5d, State: Initialized, Role: FOLLOWER
I20260620 18:35:39.388706 24305 ts_tablet_manager.cc:1397] T ede9baf238e24b538e3d0a2459a8826f P c34aba1d518346b0937a687c4f9e9564: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.004s
I20260620 18:35:39.388723 24307 consensus_queue.cc:260] T ede9baf238e24b538e3d0a2459a8826f P 2fdb54ee16854fcbbb83f8a380d57e5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fdb54ee16854fcbbb83f8a380d57e5d" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44765 } } peers { permanent_uuid: "82ee0b3ce78a4e2c996a42b9644afd48" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 37905 } } peers { permanent_uuid: "c34aba1d518346b0937a687c4f9e9564" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 39949 } }
I20260620 18:35:39.389335 24307 ts_tablet_manager.cc:1428] T ede9baf238e24b538e3d0a2459a8826f P 2fdb54ee16854fcbbb83f8a380d57e5d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260620 18:35:39.389680 24305 raft_consensus.cc:348] T ede9baf238e24b538e3d0a2459a8826f P c34aba1d518346b0937a687c4f9e9564 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fdb54ee16854fcbbb83f8a380d57e5d" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44765 } } peers { permanent_uuid: "82ee0b3ce78a4e2c996a42b9644afd48" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 37905 } } peers { permanent_uuid: "c34aba1d518346b0937a687c4f9e9564" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 39949 } }
I20260620 18:35:39.389901 24305 raft_consensus.cc:374] T ede9baf238e24b538e3d0a2459a8826f P c34aba1d518346b0937a687c4f9e9564 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:35:39.390002 24305 raft_consensus.cc:729] T ede9baf238e24b538e3d0a2459a8826f P c34aba1d518346b0937a687c4f9e9564 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c34aba1d518346b0937a687c4f9e9564, State: Initialized, Role: FOLLOWER
I20260620 18:35:39.390259 24305 consensus_queue.cc:260] T ede9baf238e24b538e3d0a2459a8826f P c34aba1d518346b0937a687c4f9e9564 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fdb54ee16854fcbbb83f8a380d57e5d" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44765 } } peers { permanent_uuid: "82ee0b3ce78a4e2c996a42b9644afd48" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 37905 } } peers { permanent_uuid: "c34aba1d518346b0937a687c4f9e9564" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 39949 } }
I20260620 18:35:39.391312 24305 ts_tablet_manager.cc:1428] T ede9baf238e24b538e3d0a2459a8826f P c34aba1d518346b0937a687c4f9e9564: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 18:35:39.475665 24311 raft_consensus.cc:482] T ede9baf238e24b538e3d0a2459a8826f P c34aba1d518346b0937a687c4f9e9564 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:35:39.475905 24311 raft_consensus.cc:504] T ede9baf238e24b538e3d0a2459a8826f P c34aba1d518346b0937a687c4f9e9564 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fdb54ee16854fcbbb83f8a380d57e5d" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44765 } } peers { permanent_uuid: "82ee0b3ce78a4e2c996a42b9644afd48" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 37905 } } peers { permanent_uuid: "c34aba1d518346b0937a687c4f9e9564" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 39949 } }
I20260620 18:35:39.477774 24311 leader_election.cc:288] T ede9baf238e24b538e3d0a2459a8826f P c34aba1d518346b0937a687c4f9e9564 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2fdb54ee16854fcbbb83f8a380d57e5d (127.22.37.1:44765), 82ee0b3ce78a4e2c996a42b9644afd48 (127.22.37.2:37905)
I20260620 18:35:39.485958 24302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:39.478615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.3:39129 (local address 127.22.37.2:37905)
0620 18:35:39.478791 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:39.478797 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:39.479349 (+ 552us) server_negotiation.cc:407] Connection header received
0620 18:35:39.479639 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:39.479647 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:39.479732 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:39.479822 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:35:39.480304 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:39.481239 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:39.483218 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:39.484452 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:39.484809 (+ 357us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:39.484916 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:39.485502 (+ 586us) server_negotiation.cc:299] Negotiation successful
0620 18:35:39.485648 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 18:35:39.486147 24304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:39.478178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.3:33627 (local address 127.22.37.1:44765)
0620 18:35:39.478361 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:39.478367 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:39.479713 (+ 1346us) server_negotiation.cc:407] Connection header received
0620 18:35:39.479942 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:39.479947 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:39.480026 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:39.480075 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:35:39.480811 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:39.482018 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:39.483896 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:39.484883 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:39.485041 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:39.485145 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:39.485712 (+ 567us) server_negotiation.cc:299] Negotiation successful
0620 18:35:39.485845 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 18:35:39.486790 24334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:39.477998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.2:37905 (local address 127.22.37.3:39129)
0620 18:35:39.479238 (+ 1240us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:39.479261 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:39.479465 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:39.479869 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:39.479873 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:39.479913 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:35:39.480133 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:39.480146 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:39.481397 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:39.481402 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:39.483052 (+ 1650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:39.483066 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:39.485032 (+ 1966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:39.485039 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:39.485267 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:39.485306 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 18:35:39.486244 (+ 938us) client_negotiation.cc:240] Negotiation successful
0620 18:35:39.486398 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1132,"thread_start_us":985,"threads_started":1}
I20260620 18:35:39.486795 24335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:39.477998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.1:44765 (local address 127.22.37.3:33627)
0620 18:35:39.479587 (+ 1589us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:39.479610 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:39.479747 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:39.480223 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:39.480249 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:39.480322 (+ 73us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:35:39.480640 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:39.480652 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:39.482155 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:39.482161 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:39.483737 (+ 1576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:39.483747 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:39.485310 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:39.485316 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:39.485530 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:39.485566 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 18:35:39.486386 (+ 820us) client_negotiation.cc:240] Negotiation successful
0620 18:35:39.486509 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1478,"mutex_wait_us":170,"thread_start_us":1119,"threads_started":1}
I20260620 18:35:39.487984 24097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ede9baf238e24b538e3d0a2459a8826f" candidate_uuid: "c34aba1d518346b0937a687c4f9e9564" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82ee0b3ce78a4e2c996a42b9644afd48" is_pre_election: true
I20260620 18:35:39.488025 23963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ede9baf238e24b538e3d0a2459a8826f" candidate_uuid: "c34aba1d518346b0937a687c4f9e9564" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fdb54ee16854fcbbb83f8a380d57e5d" is_pre_election: true
I20260620 18:35:39.488270 24097 raft_consensus.cc:2454] T ede9baf238e24b538e3d0a2459a8826f P 82ee0b3ce78a4e2c996a42b9644afd48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c34aba1d518346b0937a687c4f9e9564 in term 0.
I20260620 18:35:39.488512 23963 raft_consensus.cc:2454] T ede9baf238e24b538e3d0a2459a8826f P 2fdb54ee16854fcbbb83f8a380d57e5d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c34aba1d518346b0937a687c4f9e9564 in term 0.
I20260620 18:35:39.488762 24170 leader_election.cc:302] T ede9baf238e24b538e3d0a2459a8826f P c34aba1d518346b0937a687c4f9e9564 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82ee0b3ce78a4e2c996a42b9644afd48, c34aba1d518346b0937a687c4f9e9564; no voters:
I20260620 18:35:39.489189 24311 raft_consensus.cc:2781] T ede9baf238e24b538e3d0a2459a8826f P c34aba1d518346b0937a687c4f9e9564 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:35:39.489317 24311 raft_consensus.cc:482] T ede9baf238e24b538e3d0a2459a8826f P c34aba1d518346b0937a687c4f9e9564 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:35:39.489452 24311 raft_consensus.cc:3037] T ede9baf238e24b538e3d0a2459a8826f P c34aba1d518346b0937a687c4f9e9564 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:35:39.491524 24311 raft_consensus.cc:504] T ede9baf238e24b538e3d0a2459a8826f P c34aba1d518346b0937a687c4f9e9564 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fdb54ee16854fcbbb83f8a380d57e5d" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44765 } } peers { permanent_uuid: "82ee0b3ce78a4e2c996a42b9644afd48" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 37905 } } peers { permanent_uuid: "c34aba1d518346b0937a687c4f9e9564" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 39949 } }
I20260620 18:35:39.492127 24311 leader_election.cc:288] T ede9baf238e24b538e3d0a2459a8826f P c34aba1d518346b0937a687c4f9e9564 [CANDIDATE]: Term 1 election: Requested vote from peers 2fdb54ee16854fcbbb83f8a380d57e5d (127.22.37.1:44765), 82ee0b3ce78a4e2c996a42b9644afd48 (127.22.37.2:37905)
I20260620 18:35:39.492580 23963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ede9baf238e24b538e3d0a2459a8826f" candidate_uuid: "c34aba1d518346b0937a687c4f9e9564" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fdb54ee16854fcbbb83f8a380d57e5d"
I20260620 18:35:39.492671 24097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ede9baf238e24b538e3d0a2459a8826f" candidate_uuid: "c34aba1d518346b0937a687c4f9e9564" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82ee0b3ce78a4e2c996a42b9644afd48"
I20260620 18:35:39.492767 23963 raft_consensus.cc:3037] T ede9baf238e24b538e3d0a2459a8826f P 2fdb54ee16854fcbbb83f8a380d57e5d [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:35:39.492883 24097 raft_consensus.cc:3037] T ede9baf238e24b538e3d0a2459a8826f P 82ee0b3ce78a4e2c996a42b9644afd48 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:35:39.495065 23963 raft_consensus.cc:2454] T ede9baf238e24b538e3d0a2459a8826f P 2fdb54ee16854fcbbb83f8a380d57e5d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c34aba1d518346b0937a687c4f9e9564 in term 1.
I20260620 18:35:39.495467 24169 leader_election.cc:302] T ede9baf238e24b538e3d0a2459a8826f P c34aba1d518346b0937a687c4f9e9564 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fdb54ee16854fcbbb83f8a380d57e5d, c34aba1d518346b0937a687c4f9e9564; no voters:
I20260620 18:35:39.495736 24311 raft_consensus.cc:2781] T ede9baf238e24b538e3d0a2459a8826f P c34aba1d518346b0937a687c4f9e9564 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:35:39.495839 24097 raft_consensus.cc:2454] T ede9baf238e24b538e3d0a2459a8826f P 82ee0b3ce78a4e2c996a42b9644afd48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c34aba1d518346b0937a687c4f9e9564 in term 1.
I20260620 18:35:39.496259 24311 raft_consensus.cc:686] T ede9baf238e24b538e3d0a2459a8826f P c34aba1d518346b0937a687c4f9e9564 [term 1 LEADER]: Becoming Leader. State: Replica: c34aba1d518346b0937a687c4f9e9564, State: Running, Role: LEADER
I20260620 18:35:39.496677 24311 consensus_queue.cc:237] T ede9baf238e24b538e3d0a2459a8826f P c34aba1d518346b0937a687c4f9e9564 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fdb54ee16854fcbbb83f8a380d57e5d" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44765 } } peers { permanent_uuid: "82ee0b3ce78a4e2c996a42b9644afd48" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 37905 } } peers { permanent_uuid: "c34aba1d518346b0937a687c4f9e9564" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 39949 } }
I20260620 18:35:39.500617 23800 catalog_manager.cc:5495] T ede9baf238e24b538e3d0a2459a8826f P c34aba1d518346b0937a687c4f9e9564 reported cstate change: term changed from 0 to 1, leader changed from <none> to c34aba1d518346b0937a687c4f9e9564 (127.22.37.3). New cstate: current_term: 1 leader_uuid: "c34aba1d518346b0937a687c4f9e9564" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fdb54ee16854fcbbb83f8a380d57e5d" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82ee0b3ce78a4e2c996a42b9644afd48" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 37905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c34aba1d518346b0937a687c4f9e9564" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 39949 } health_report { overall_health: HEALTHY } } }
Jun 20 18:35:39 dist-test-slave-5lgp krb5kdc[23756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980537, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.62@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM
I20260620 18:35:39.554729 24344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:39.539700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41562 (local address 127.22.37.62:37885)
0620 18:35:39.540848 (+ 1148us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:39.540853 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:39.540871 (+ 18us) server_negotiation.cc:407] Connection header received
0620 18:35:39.540919 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:39.540922 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:39.540992 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:39.541085 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:39.542529 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:39.543345 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:39.544612 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:39.545063 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:39.545186 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:39.548862 (+ 3676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:39.548867 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:39.548871 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:39.548901 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:39.551813 (+ 2912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:39.552629 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:39.552637 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:39.552640 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:39.552771 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:39.553132 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:39.553139 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:39.553143 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:39.553482 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:39.553649 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:39.554077 (+ 428us) server_negotiation.cc:299] Negotiation successful
0620 18:35:39.554298 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1040,"thread_start_us":963,"threads_started":1}
I20260620 18:35:39.554792 24343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:39.539539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:37885 (local address 127.0.0.1:41562)
0620 18:35:39.540230 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:39.540254 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:39.540405 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:39.541157 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:39.541163 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:39.542086 (+ 923us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:39.542366 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:39.542379 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:39.543503 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:39.543510 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:39.544439 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:39.544450 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:39.545203 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:39.545208 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:39.545414 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:39.546209 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:35:39.546227 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:39.548655 (+ 2428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:35:39.552001 (+ 3346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:39.552018 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:39.552026 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:39.552510 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:39.552873 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:39.552879 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:39.552883 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:39.553033 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:39.553629 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:35:39.553637 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:35:39.553809 (+ 172us) client_negotiation.cc:769] Sending connection context
0620 18:35:39.553992 (+ 183us) client_negotiation.cc:240] Negotiation successful
0620 18:35:39.554416 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"mutex_wait_us":143,"thread_start_us":297,"threads_started":1}
I20260620 18:35:39.563683 23800 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.22.37.62@KRBTEST.COM'} at 127.0.0.1:41562:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 18:35:39.564697 23800 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:35:39.571571 23938 tablet_service.cc:1467] Processing CreateTablet for tablet 0c72728a09e642018965c6d73296b58a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cbd62659b178469fae36215639ddc041]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 18:35:39.571928 24216 tablet_service.cc:1467] Processing CreateTablet for tablet 0c72728a09e642018965c6d73296b58a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cbd62659b178469fae36215639ddc041]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 18:35:39.572153 23938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c72728a09e642018965c6d73296b58a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:35:39.572484 24216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c72728a09e642018965c6d73296b58a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:35:39.577917 24076 tablet_service.cc:1467] Processing CreateTablet for tablet 0c72728a09e642018965c6d73296b58a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cbd62659b178469fae36215639ddc041]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 18:35:39.578454 24076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c72728a09e642018965c6d73296b58a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:35:39.579320 24307 tablet_bootstrap.cc:492] T 0c72728a09e642018965c6d73296b58a P 2fdb54ee16854fcbbb83f8a380d57e5d: Bootstrap starting.
I20260620 18:35:39.579965 24305 tablet_bootstrap.cc:492] T 0c72728a09e642018965c6d73296b58a P c34aba1d518346b0937a687c4f9e9564: Bootstrap starting.
I20260620 18:35:39.581110 24307 tablet_bootstrap.cc:654] T 0c72728a09e642018965c6d73296b58a P 2fdb54ee16854fcbbb83f8a380d57e5d: Neither blocks nor log segments found. Creating new log.
I20260620 18:35:39.582046 24305 tablet_bootstrap.cc:654] T 0c72728a09e642018965c6d73296b58a P c34aba1d518346b0937a687c4f9e9564: Neither blocks nor log segments found. Creating new log.
I20260620 18:35:39.583927 24307 tablet_bootstrap.cc:492] T 0c72728a09e642018965c6d73296b58a P 2fdb54ee16854fcbbb83f8a380d57e5d: No bootstrap required, opened a new log
I20260620 18:35:39.584129 24307 ts_tablet_manager.cc:1397] T 0c72728a09e642018965c6d73296b58a P 2fdb54ee16854fcbbb83f8a380d57e5d: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260620 18:35:39.584982 24308 tablet_bootstrap.cc:492] T 0c72728a09e642018965c6d73296b58a P 82ee0b3ce78a4e2c996a42b9644afd48: Bootstrap starting.
I20260620 18:35:39.585067 24307 raft_consensus.cc:348] T 0c72728a09e642018965c6d73296b58a P 2fdb54ee16854fcbbb83f8a380d57e5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fdb54ee16854fcbbb83f8a380d57e5d" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44765 } } peers { permanent_uuid: "82ee0b3ce78a4e2c996a42b9644afd48" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 37905 } } peers { permanent_uuid: "c34aba1d518346b0937a687c4f9e9564" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 39949 } }
I20260620 18:35:39.585307 24307 raft_consensus.cc:374] T 0c72728a09e642018965c6d73296b58a P 2fdb54ee16854fcbbb83f8a380d57e5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:35:39.585408 24307 raft_consensus.cc:729] T 0c72728a09e642018965c6d73296b58a P 2fdb54ee16854fcbbb83f8a380d57e5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fdb54ee16854fcbbb83f8a380d57e5d, State: Initialized, Role: FOLLOWER
I20260620 18:35:39.585745 24307 consensus_queue.cc:260] T 0c72728a09e642018965c6d73296b58a P 2fdb54ee16854fcbbb83f8a380d57e5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fdb54ee16854fcbbb83f8a380d57e5d" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44765 } } peers { permanent_uuid: "82ee0b3ce78a4e2c996a42b9644afd48" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 37905 } } peers { permanent_uuid: "c34aba1d518346b0937a687c4f9e9564" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 39949 } }
I20260620 18:35:39.586634 24308 tablet_bootstrap.cc:654] T 0c72728a09e642018965c6d73296b58a P 82ee0b3ce78a4e2c996a42b9644afd48: Neither blocks nor log segments found. Creating new log.
I20260620 18:35:39.587826 24307 ts_tablet_manager.cc:1428] T 0c72728a09e642018965c6d73296b58a P 2fdb54ee16854fcbbb83f8a380d57e5d: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260620 18:35:39.586347 24312 tablet_replica.cc:439] T 0c72728a09e642018965c6d73296b58a P 2fdb54ee16854fcbbb83f8a380d57e5d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fdb54ee16854fcbbb83f8a380d57e5d" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44765 } } peers { permanent_uuid: "82ee0b3ce78a4e2c996a42b9644afd48" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 37905 } } peers { permanent_uuid: "c34aba1d518346b0937a687c4f9e9564" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 39949 } } }
I20260620 18:35:39.588232 24312 tablet_replica.cc:442] T 0c72728a09e642018965c6d73296b58a P 2fdb54ee16854fcbbb83f8a380d57e5d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:35:39.588821 24308 tablet_bootstrap.cc:492] T 0c72728a09e642018965c6d73296b58a P 82ee0b3ce78a4e2c996a42b9644afd48: No bootstrap required, opened a new log
I20260620 18:35:39.588817 24305 tablet_bootstrap.cc:492] T 0c72728a09e642018965c6d73296b58a P c34aba1d518346b0937a687c4f9e9564: No bootstrap required, opened a new log
I20260620 18:35:39.588961 24308 ts_tablet_manager.cc:1397] T 0c72728a09e642018965c6d73296b58a P 82ee0b3ce78a4e2c996a42b9644afd48: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20260620 18:35:39.588965 24305 ts_tablet_manager.cc:1397] T 0c72728a09e642018965c6d73296b58a P c34aba1d518346b0937a687c4f9e9564: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.005s
I20260620 18:35:39.589684 24308 raft_consensus.cc:348] T 0c72728a09e642018965c6d73296b58a P 82ee0b3ce78a4e2c996a42b9644afd48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fdb54ee16854fcbbb83f8a380d57e5d" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44765 } } peers { permanent_uuid: "82ee0b3ce78a4e2c996a42b9644afd48" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 37905 } } peers { permanent_uuid: "c34aba1d518346b0937a687c4f9e9564" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 39949 } }
I20260620 18:35:39.589684 24305 raft_consensus.cc:348] T 0c72728a09e642018965c6d73296b58a P c34aba1d518346b0937a687c4f9e9564 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fdb54ee16854fcbbb83f8a380d57e5d" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44765 } } peers { permanent_uuid: "82ee0b3ce78a4e2c996a42b9644afd48" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 37905 } } peers { permanent_uuid: "c34aba1d518346b0937a687c4f9e9564" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 39949 } }
I20260620 18:35:39.589903 24308 raft_consensus.cc:374] T 0c72728a09e642018965c6d73296b58a P 82ee0b3ce78a4e2c996a42b9644afd48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:35:39.589910 24305 raft_consensus.cc:374] T 0c72728a09e642018965c6d73296b58a P c34aba1d518346b0937a687c4f9e9564 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:35:39.589977 24305 raft_consensus.cc:729] T 0c72728a09e642018965c6d73296b58a P c34aba1d518346b0937a687c4f9e9564 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c34aba1d518346b0937a687c4f9e9564, State: Initialized, Role: FOLLOWER
I20260620 18:35:39.589977 24308 raft_consensus.cc:729] T 0c72728a09e642018965c6d73296b58a P 82ee0b3ce78a4e2c996a42b9644afd48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82ee0b3ce78a4e2c996a42b9644afd48, State: Initialized, Role: FOLLOWER
I20260620 18:35:39.590193 24305 consensus_queue.cc:260] T 0c72728a09e642018965c6d73296b58a P c34aba1d518346b0937a687c4f9e9564 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fdb54ee16854fcbbb83f8a380d57e5d" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44765 } } peers { permanent_uuid: "82ee0b3ce78a4e2c996a42b9644afd48" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 37905 } } peers { permanent_uuid: "c34aba1d518346b0937a687c4f9e9564" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 39949 } }
I20260620 18:35:39.590225 24308 consensus_queue.cc:260] T 0c72728a09e642018965c6d73296b58a P 82ee0b3ce78a4e2c996a42b9644afd48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fdb54ee16854fcbbb83f8a380d57e5d" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44765 } } peers { permanent_uuid: "82ee0b3ce78a4e2c996a42b9644afd48" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 37905 } } peers { permanent_uuid: "c34aba1d518346b0937a687c4f9e9564" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 39949 } }
I20260620 18:35:39.590714 24305 ts_tablet_manager.cc:1428] T 0c72728a09e642018965c6d73296b58a P c34aba1d518346b0937a687c4f9e9564: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 18:35:39.590742 24308 ts_tablet_manager.cc:1428] T 0c72728a09e642018965c6d73296b58a P 82ee0b3ce78a4e2c996a42b9644afd48: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 18:35:39.591893 24313 tablet_replica.cc:439] T 0c72728a09e642018965c6d73296b58a P 82ee0b3ce78a4e2c996a42b9644afd48: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fdb54ee16854fcbbb83f8a380d57e5d" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44765 } } peers { permanent_uuid: "82ee0b3ce78a4e2c996a42b9644afd48" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 37905 } } peers { permanent_uuid: "c34aba1d518346b0937a687c4f9e9564" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 39949 } } }
I20260620 18:35:39.592242 24313 tablet_replica.cc:442] T 0c72728a09e642018965c6d73296b58a P 82ee0b3ce78a4e2c996a42b9644afd48: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:35:39.592519 24311 tablet_replica.cc:439] T 0c72728a09e642018965c6d73296b58a P c34aba1d518346b0937a687c4f9e9564: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fdb54ee16854fcbbb83f8a380d57e5d" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44765 } } peers { permanent_uuid: "82ee0b3ce78a4e2c996a42b9644afd48" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 37905 } } peers { permanent_uuid: "c34aba1d518346b0937a687c4f9e9564" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 39949 } } }
I20260620 18:35:39.592775 24311 tablet_replica.cc:442] T 0c72728a09e642018965c6d73296b58a P c34aba1d518346b0937a687c4f9e9564: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:35:39.619288 24311 raft_consensus.cc:482] T 0c72728a09e642018965c6d73296b58a P c34aba1d518346b0937a687c4f9e9564 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:35:39.619503 24311 raft_consensus.cc:504] T 0c72728a09e642018965c6d73296b58a P c34aba1d518346b0937a687c4f9e9564 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fdb54ee16854fcbbb83f8a380d57e5d" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44765 } } peers { permanent_uuid: "82ee0b3ce78a4e2c996a42b9644afd48" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 37905 } } peers { permanent_uuid: "c34aba1d518346b0937a687c4f9e9564" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 39949 } }
I20260620 18:35:39.620288 24311 leader_election.cc:288] T 0c72728a09e642018965c6d73296b58a P c34aba1d518346b0937a687c4f9e9564 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2fdb54ee16854fcbbb83f8a380d57e5d (127.22.37.1:44765), 82ee0b3ce78a4e2c996a42b9644afd48 (127.22.37.2:37905)
I20260620 18:35:39.620859 23963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c72728a09e642018965c6d73296b58a" candidate_uuid: "c34aba1d518346b0937a687c4f9e9564" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fdb54ee16854fcbbb83f8a380d57e5d" is_pre_election: true
I20260620 18:35:39.621009 24097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c72728a09e642018965c6d73296b58a" candidate_uuid: "c34aba1d518346b0937a687c4f9e9564" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82ee0b3ce78a4e2c996a42b9644afd48" is_pre_election: true
I20260620 18:35:39.621117 23963 raft_consensus.cc:2454] T 0c72728a09e642018965c6d73296b58a P 2fdb54ee16854fcbbb83f8a380d57e5d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c34aba1d518346b0937a687c4f9e9564 in term 0.
I20260620 18:35:39.621294 24097 raft_consensus.cc:2454] T 0c72728a09e642018965c6d73296b58a P 82ee0b3ce78a4e2c996a42b9644afd48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c34aba1d518346b0937a687c4f9e9564 in term 0.
I20260620 18:35:39.621513 24169 leader_election.cc:302] T 0c72728a09e642018965c6d73296b58a P c34aba1d518346b0937a687c4f9e9564 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fdb54ee16854fcbbb83f8a380d57e5d, c34aba1d518346b0937a687c4f9e9564; no voters:
I20260620 18:35:39.621809 24311 raft_consensus.cc:2781] T 0c72728a09e642018965c6d73296b58a P c34aba1d518346b0937a687c4f9e9564 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:35:39.621903 24311 raft_consensus.cc:482] T 0c72728a09e642018965c6d73296b58a P c34aba1d518346b0937a687c4f9e9564 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:35:39.622025 24311 raft_consensus.cc:3037] T 0c72728a09e642018965c6d73296b58a P c34aba1d518346b0937a687c4f9e9564 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:35:39.624513 24311 raft_consensus.cc:504] T 0c72728a09e642018965c6d73296b58a P c34aba1d518346b0937a687c4f9e9564 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fdb54ee16854fcbbb83f8a380d57e5d" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44765 } } peers { permanent_uuid: "82ee0b3ce78a4e2c996a42b9644afd48" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 37905 } } peers { permanent_uuid: "c34aba1d518346b0937a687c4f9e9564" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 39949 } }
I20260620 18:35:39.625095 24311 leader_election.cc:288] T 0c72728a09e642018965c6d73296b58a P c34aba1d518346b0937a687c4f9e9564 [CANDIDATE]: Term 1 election: Requested vote from peers 2fdb54ee16854fcbbb83f8a380d57e5d (127.22.37.1:44765), 82ee0b3ce78a4e2c996a42b9644afd48 (127.22.37.2:37905)
I20260620 18:35:39.625440 23963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c72728a09e642018965c6d73296b58a" candidate_uuid: "c34aba1d518346b0937a687c4f9e9564" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fdb54ee16854fcbbb83f8a380d57e5d"
I20260620 18:35:39.625540 24097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c72728a09e642018965c6d73296b58a" candidate_uuid: "c34aba1d518346b0937a687c4f9e9564" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82ee0b3ce78a4e2c996a42b9644afd48"
I20260620 18:35:39.625671 23963 raft_consensus.cc:3037] T 0c72728a09e642018965c6d73296b58a P 2fdb54ee16854fcbbb83f8a380d57e5d [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:35:39.625771 24097 raft_consensus.cc:3037] T 0c72728a09e642018965c6d73296b58a P 82ee0b3ce78a4e2c996a42b9644afd48 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:35:39.627761 23963 raft_consensus.cc:2454] T 0c72728a09e642018965c6d73296b58a P 2fdb54ee16854fcbbb83f8a380d57e5d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c34aba1d518346b0937a687c4f9e9564 in term 1.
I20260620 18:35:39.628140 24169 leader_election.cc:302] T 0c72728a09e642018965c6d73296b58a P c34aba1d518346b0937a687c4f9e9564 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fdb54ee16854fcbbb83f8a380d57e5d, c34aba1d518346b0937a687c4f9e9564; no voters:
I20260620 18:35:39.628374 24311 raft_consensus.cc:2781] T 0c72728a09e642018965c6d73296b58a P c34aba1d518346b0937a687c4f9e9564 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:35:39.628571 24311 raft_consensus.cc:686] T 0c72728a09e642018965c6d73296b58a P c34aba1d518346b0937a687c4f9e9564 [term 1 LEADER]: Becoming Leader. State: Replica: c34aba1d518346b0937a687c4f9e9564, State: Running, Role: LEADER
I20260620 18:35:39.628638 24097 raft_consensus.cc:2454] T 0c72728a09e642018965c6d73296b58a P 82ee0b3ce78a4e2c996a42b9644afd48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c34aba1d518346b0937a687c4f9e9564 in term 1.
I20260620 18:35:39.628801 24311 consensus_queue.cc:237] T 0c72728a09e642018965c6d73296b58a P c34aba1d518346b0937a687c4f9e9564 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fdb54ee16854fcbbb83f8a380d57e5d" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44765 } } peers { permanent_uuid: "82ee0b3ce78a4e2c996a42b9644afd48" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 37905 } } peers { permanent_uuid: "c34aba1d518346b0937a687c4f9e9564" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 39949 } }
I20260620 18:35:39.630473 24336 tablet_replica.cc:439] T 0c72728a09e642018965c6d73296b58a P c34aba1d518346b0937a687c4f9e9564: TxnStatusTablet state changed. Reason: New leader c34aba1d518346b0937a687c4f9e9564. Latest consensus state: current_term: 1 leader_uuid: "c34aba1d518346b0937a687c4f9e9564" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fdb54ee16854fcbbb83f8a380d57e5d" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44765 } } peers { permanent_uuid: "82ee0b3ce78a4e2c996a42b9644afd48" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 37905 } } peers { permanent_uuid: "c34aba1d518346b0937a687c4f9e9564" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 39949 } } }
I20260620 18:35:39.630817 24336 tablet_replica.cc:442] T 0c72728a09e642018965c6d73296b58a P c34aba1d518346b0937a687c4f9e9564: This TxnStatusTablet replica's current role is: LEADER
I20260620 18:35:39.632899 23799 catalog_manager.cc:5495] T 0c72728a09e642018965c6d73296b58a P c34aba1d518346b0937a687c4f9e9564 reported cstate change: term changed from 0 to 1, leader changed from <none> to c34aba1d518346b0937a687c4f9e9564 (127.22.37.3). New cstate: current_term: 1 leader_uuid: "c34aba1d518346b0937a687c4f9e9564" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fdb54ee16854fcbbb83f8a380d57e5d" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82ee0b3ce78a4e2c996a42b9644afd48" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 37905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c34aba1d518346b0937a687c4f9e9564" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 39949 } health_report { overall_health: HEALTHY } } }
I20260620 18:35:39.633911 24349 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 18:35:39.634133 24349 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 18:35:39.661305 24343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:39.653800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.3:39949 (local address 127.0.0.1:52070)
0620 18:35:39.654025 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:39.654041 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:39.654143 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:39.654673 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:39.654678 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:39.654694 (+ 16us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 18:35:39.654871 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:39.654883 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:39.656483 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:39.656489 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:39.658208 (+ 1719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:39.658223 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:39.659327 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:39.659334 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:39.659558 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:39.659630 (+ 72us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 18:35:39.660703 (+ 1073us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 18:35:39.660732 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 18:35:39.660822 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 18:35:39.660969 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 18:35:39.661710 24300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:39.653868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52070 (local address 127.22.37.3:39949)
0620 18:35:39.654156 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:39.654165 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:39.654189 (+ 24us) server_negotiation.cc:407] Connection header received
0620 18:35:39.654305 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:39.654311 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:39.654446 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:39.654617 (+ 171us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:35:39.655017 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:39.656331 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:39.658400 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:39.659072 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:39.659199 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:39.659839 (+ 640us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:35:39.660584 (+ 745us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:35:39.660734 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:39.661265 (+ 531us) server_negotiation.cc:299] Negotiation successful
0620 18:35:39.661347 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167}
I20260620 18:35:39.666289 24097 raft_consensus.cc:1261] T 0c72728a09e642018965c6d73296b58a P 82ee0b3ce78a4e2c996a42b9644afd48 [term 1 FOLLOWER]: Refusing update from remote peer c34aba1d518346b0937a687c4f9e9564: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:35:39.666890 23963 raft_consensus.cc:1261] T 0c72728a09e642018965c6d73296b58a P 2fdb54ee16854fcbbb83f8a380d57e5d [term 1 FOLLOWER]: Refusing update from remote peer c34aba1d518346b0937a687c4f9e9564: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:35:39.667771 24311 consensus_queue.cc:1035] T 0c72728a09e642018965c6d73296b58a P c34aba1d518346b0937a687c4f9e9564 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fdb54ee16854fcbbb83f8a380d57e5d" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:35:39.668128 24336 consensus_queue.cc:1035] T 0c72728a09e642018965c6d73296b58a P c34aba1d518346b0937a687c4f9e9564 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82ee0b3ce78a4e2c996a42b9644afd48" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 37905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:35:39.678289 24313 tablet_replica.cc:439] T 0c72728a09e642018965c6d73296b58a P 82ee0b3ce78a4e2c996a42b9644afd48: TxnStatusTablet state changed. Reason: New leader c34aba1d518346b0937a687c4f9e9564. Latest consensus state: current_term: 1 leader_uuid: "c34aba1d518346b0937a687c4f9e9564" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fdb54ee16854fcbbb83f8a380d57e5d" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44765 } } peers { permanent_uuid: "82ee0b3ce78a4e2c996a42b9644afd48" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 37905 } } peers { permanent_uuid: "c34aba1d518346b0937a687c4f9e9564" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 39949 } } }
I20260620 18:35:39.678664 24313 tablet_replica.cc:442] T 0c72728a09e642018965c6d73296b58a P 82ee0b3ce78a4e2c996a42b9644afd48: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:35:39.683233 24311 tablet_replica.cc:439] T 0c72728a09e642018965c6d73296b58a P c34aba1d518346b0937a687c4f9e9564: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c34aba1d518346b0937a687c4f9e9564" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fdb54ee16854fcbbb83f8a380d57e5d" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44765 } } peers { permanent_uuid: "82ee0b3ce78a4e2c996a42b9644afd48" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 37905 } } peers { permanent_uuid: "c34aba1d518346b0937a687c4f9e9564" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 39949 } } }
I20260620 18:35:39.683547 24311 tablet_replica.cc:442] T 0c72728a09e642018965c6d73296b58a P c34aba1d518346b0937a687c4f9e9564: This TxnStatusTablet replica's current role is: LEADER
I20260620 18:35:39.685051 24313 tablet_replica.cc:439] T 0c72728a09e642018965c6d73296b58a P 82ee0b3ce78a4e2c996a42b9644afd48: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c34aba1d518346b0937a687c4f9e9564" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fdb54ee16854fcbbb83f8a380d57e5d" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44765 } } peers { permanent_uuid: "82ee0b3ce78a4e2c996a42b9644afd48" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 37905 } } peers { permanent_uuid: "c34aba1d518346b0937a687c4f9e9564" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 39949 } } }
I20260620 18:35:39.685281 24313 tablet_replica.cc:442] T 0c72728a09e642018965c6d73296b58a P 82ee0b3ce78a4e2c996a42b9644afd48: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:35:39.696563 24312 tablet_replica.cc:439] T 0c72728a09e642018965c6d73296b58a P 2fdb54ee16854fcbbb83f8a380d57e5d: TxnStatusTablet state changed. Reason: New leader c34aba1d518346b0937a687c4f9e9564. Latest consensus state: current_term: 1 leader_uuid: "c34aba1d518346b0937a687c4f9e9564" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fdb54ee16854fcbbb83f8a380d57e5d" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44765 } } peers { permanent_uuid: "82ee0b3ce78a4e2c996a42b9644afd48" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 37905 } } peers { permanent_uuid: "c34aba1d518346b0937a687c4f9e9564" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 39949 } } }
I20260620 18:35:39.696887 24312 tablet_replica.cc:442] T 0c72728a09e642018965c6d73296b58a P 2fdb54ee16854fcbbb83f8a380d57e5d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:35:39.699362 24311 tablet_replica.cc:439] T 0c72728a09e642018965c6d73296b58a P c34aba1d518346b0937a687c4f9e9564: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c34aba1d518346b0937a687c4f9e9564" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fdb54ee16854fcbbb83f8a380d57e5d" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44765 } } peers { permanent_uuid: "82ee0b3ce78a4e2c996a42b9644afd48" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 37905 } } peers { permanent_uuid: "c34aba1d518346b0937a687c4f9e9564" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 39949 } } }
I20260620 18:35:39.700098 24311 tablet_replica.cc:442] T 0c72728a09e642018965c6d73296b58a P c34aba1d518346b0937a687c4f9e9564: This TxnStatusTablet replica's current role is: LEADER
I20260620 18:35:39.701645 24300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:39.694075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52086 (local address 127.22.37.3:39949)
0620 18:35:39.694261 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:39.694267 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:39.694286 (+ 19us) server_negotiation.cc:407] Connection header received
0620 18:35:39.694354 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:39.694359 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:39.694460 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:39.694569 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:35:39.695127 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:39.696239 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:39.698308 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:39.699141 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:39.699279 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:39.699944 (+ 665us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:35:39.700090 (+ 146us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:35:39.700190 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:39.701226 (+ 1036us) server_negotiation.cc:299] Negotiation successful
0620 18:35:39.701315 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 18:35:39.705055 24312 tablet_replica.cc:439] T 0c72728a09e642018965c6d73296b58a P 2fdb54ee16854fcbbb83f8a380d57e5d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c34aba1d518346b0937a687c4f9e9564" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fdb54ee16854fcbbb83f8a380d57e5d" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44765 } } peers { permanent_uuid: "82ee0b3ce78a4e2c996a42b9644afd48" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 37905 } } peers { permanent_uuid: "c34aba1d518346b0937a687c4f9e9564" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 39949 } } }
I20260620 18:35:39.705338 24312 tablet_replica.cc:442] T 0c72728a09e642018965c6d73296b58a P 2fdb54ee16854fcbbb83f8a380d57e5d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:35:39.722554 24300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:39.714831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.3:36241 (local address 127.22.37.3:39949)
0620 18:35:39.715018 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:39.715026 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:39.715236 (+ 210us) server_negotiation.cc:407] Connection header received
0620 18:35:39.715913 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:39.715919 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:39.716004 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:39.716104 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:35:39.716678 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:39.717861 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:39.720314 (+ 2453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:39.721073 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:39.721201 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:39.721747 (+ 546us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:35:39.721896 (+ 149us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:35:39.721983 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:39.722141 (+ 158us) server_negotiation.cc:299] Negotiation successful
0620 18:35:39.722216 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 18:35:39.722895 24356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:39.714483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.3:39949 (local address 127.22.37.3:36241)
0620 18:35:39.715146 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:39.715166 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:39.715342 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:39.716131 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:39.716137 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:39.716149 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 18:35:39.716498 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:39.716509 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:39.718025 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:39.718030 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:39.720141 (+ 2111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:39.720156 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:39.721264 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:39.721271 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:39.721501 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:39.721569 (+ 68us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 18:35:39.722001 (+ 432us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 18:35:39.722012 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 18:35:39.722111 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 18:35:39.722274 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":548,"threads_started":1}
I20260620 18:35:39.732888 24097 raft_consensus.cc:1261] T ede9baf238e24b538e3d0a2459a8826f P 82ee0b3ce78a4e2c996a42b9644afd48 [term 1 FOLLOWER]: Refusing update from remote peer c34aba1d518346b0937a687c4f9e9564: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:35:39.733559 24311 consensus_queue.cc:1035] T ede9baf238e24b538e3d0a2459a8826f P c34aba1d518346b0937a687c4f9e9564 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82ee0b3ce78a4e2c996a42b9644afd48" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 37905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:35:39.734653 23963 raft_consensus.cc:1261] T ede9baf238e24b538e3d0a2459a8826f P 2fdb54ee16854fcbbb83f8a380d57e5d [term 1 FOLLOWER]: Refusing update from remote peer c34aba1d518346b0937a687c4f9e9564: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:35:39.735217 24311 consensus_queue.cc:1035] T ede9baf238e24b538e3d0a2459a8826f P c34aba1d518346b0937a687c4f9e9564 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fdb54ee16854fcbbb83f8a380d57e5d" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:35:39.834508 23799 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41556:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:35:39.834836 23799 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41556:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:35:39.838491 23799 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f72af03185aa4be6bfacf552ced6b7a4: Sending DeleteTablet for 3 replicas of tablet ede9baf238e24b538e3d0a2459a8826f
I20260620 18:35:39.839382 24076 tablet_service.cc:1514] Processing DeleteTablet for tablet ede9baf238e24b538e3d0a2459a8826f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:35:39 UTC) from {username='kudu', principal='kudu/127.22.37.62@KRBTEST.COM'} at 127.0.0.1:55758
I20260620 18:35:39.839390 23938 tablet_service.cc:1514] Processing DeleteTablet for tablet ede9baf238e24b538e3d0a2459a8826f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:35:39 UTC) from {username='kudu', principal='kudu/127.22.37.62@KRBTEST.COM'} at 127.0.0.1:56080
I20260620 18:35:39.839675 24329 tablet_replica.cc:330] T ede9baf238e24b538e3d0a2459a8826f P 2fdb54ee16854fcbbb83f8a380d57e5d: stopping tablet replica
I20260620 18:35:39.839675 24330 tablet_replica.cc:330] T ede9baf238e24b538e3d0a2459a8826f P 82ee0b3ce78a4e2c996a42b9644afd48: stopping tablet replica
I20260620 18:35:39.839910 24330 raft_consensus.cc:2229] T ede9baf238e24b538e3d0a2459a8826f P 82ee0b3ce78a4e2c996a42b9644afd48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:35:39.839941 24329 raft_consensus.cc:2229] T ede9baf238e24b538e3d0a2459a8826f P 2fdb54ee16854fcbbb83f8a380d57e5d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:35:39.840124 24330 raft_consensus.cc:2258] T ede9baf238e24b538e3d0a2459a8826f P 82ee0b3ce78a4e2c996a42b9644afd48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:35:39.840147 24329 raft_consensus.cc:2258] T ede9baf238e24b538e3d0a2459a8826f P 2fdb54ee16854fcbbb83f8a380d57e5d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:35:39.840369 24215 tablet_service.cc:1514] Processing DeleteTablet for tablet ede9baf238e24b538e3d0a2459a8826f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:35:39 UTC) from {username='kudu', principal='kudu/127.22.37.62@KRBTEST.COM'} at 127.0.0.1:52056
I20260620 18:35:39.840719 24328 tablet_replica.cc:330] T ede9baf238e24b538e3d0a2459a8826f P c34aba1d518346b0937a687c4f9e9564: stopping tablet replica
I20260620 18:35:39.840941 24328 raft_consensus.cc:2229] T ede9baf238e24b538e3d0a2459a8826f P c34aba1d518346b0937a687c4f9e9564 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:35:39.841480 24328 raft_consensus.cc:2258] T ede9baf238e24b538e3d0a2459a8826f P c34aba1d518346b0937a687c4f9e9564 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:35:39.843331 24328 ts_tablet_manager.cc:1905] T ede9baf238e24b538e3d0a2459a8826f P c34aba1d518346b0937a687c4f9e9564: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:35:39.854777 24329 ts_tablet_manager.cc:1905] T ede9baf238e24b538e3d0a2459a8826f P 2fdb54ee16854fcbbb83f8a380d57e5d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:35:39.864799 24329 ts_tablet_manager.cc:1918] T ede9baf238e24b538e3d0a2459a8826f P 2fdb54ee16854fcbbb83f8a380d57e5d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:35:39.865036 24329 log.cc:1196] T ede9baf238e24b538e3d0a2459a8826f P 2fdb54ee16854fcbbb83f8a380d57e5d: Deleting WAL directory at /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980527431848-22676-0/minicluster-data/ts-0/wal/wals/ede9baf238e24b538e3d0a2459a8826f
I20260620 18:35:39.866575 24330 ts_tablet_manager.cc:1905] T ede9baf238e24b538e3d0a2459a8826f P 82ee0b3ce78a4e2c996a42b9644afd48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:35:39.876036 24330 ts_tablet_manager.cc:1918] T ede9baf238e24b538e3d0a2459a8826f P 82ee0b3ce78a4e2c996a42b9644afd48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:35:39.876247 24330 log.cc:1196] T ede9baf238e24b538e3d0a2459a8826f P 82ee0b3ce78a4e2c996a42b9644afd48: Deleting WAL directory at /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980527431848-22676-0/minicluster-data/ts-1/wal/wals/ede9baf238e24b538e3d0a2459a8826f
I20260620 18:35:39.876807 24330 ts_tablet_manager.cc:1939] T ede9baf238e24b538e3d0a2459a8826f P 82ee0b3ce78a4e2c996a42b9644afd48: Deleting consensus metadata
I20260620 18:35:39.877794 24328 ts_tablet_manager.cc:1918] T ede9baf238e24b538e3d0a2459a8826f P c34aba1d518346b0937a687c4f9e9564: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:35:39.878068 24328 log.cc:1196] T ede9baf238e24b538e3d0a2459a8826f P c34aba1d518346b0937a687c4f9e9564: Deleting WAL directory at /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980527431848-22676-0/minicluster-data/ts-2/wal/wals/ede9baf238e24b538e3d0a2459a8826f
I20260620 18:35:39.878347 23788 catalog_manager.cc:4841] TS 82ee0b3ce78a4e2c996a42b9644afd48 (127.22.37.2:37905): tablet ede9baf238e24b538e3d0a2459a8826f (table test-table [id=a7a9cc630f7b428bb49b5fd6bc40fb83]) successfully deleted
I20260620 18:35:39.878443 24329 ts_tablet_manager.cc:1939] T ede9baf238e24b538e3d0a2459a8826f P 2fdb54ee16854fcbbb83f8a380d57e5d: Deleting consensus metadata
I20260620 18:35:39.879017 24328 ts_tablet_manager.cc:1939] T ede9baf238e24b538e3d0a2459a8826f P c34aba1d518346b0937a687c4f9e9564: Deleting consensus metadata
I20260620 18:35:39.880167 23787 catalog_manager.cc:4841] TS 2fdb54ee16854fcbbb83f8a380d57e5d (127.22.37.1:44765): tablet ede9baf238e24b538e3d0a2459a8826f (table test-table [id=a7a9cc630f7b428bb49b5fd6bc40fb83]) successfully deleted
I20260620 18:35:39.881539 23785 catalog_manager.cc:4841] TS c34aba1d518346b0937a687c4f9e9564 (127.22.37.3:39949): tablet ede9baf238e24b538e3d0a2459a8826f (table test-table [id=a7a9cc630f7b428bb49b5fd6bc40fb83]) successfully deleted
Jun 20 18:35:39 dist-test-slave-5lgp krb5kdc[23756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980538, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.22.37.1@KRBTEST.COM
I20260620 18:35:39.896287 24304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:39.879914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56096 (local address 127.22.37.1:44765)
0620 18:35:39.880097 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:39.880103 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:39.882297 (+ 2194us) server_negotiation.cc:407] Connection header received
0620 18:35:39.882500 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:39.882505 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:39.882598 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:39.882700 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:39.884158 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:39.884866 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:39.886220 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:39.886627 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:39.886763 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:39.890805 (+ 4042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:39.890818 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:39.890834 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:39.890868 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:39.893652 (+ 2784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:39.894305 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:39.894312 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:39.894314 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:39.894449 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:39.894814 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:39.894819 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:39.894822 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:39.895177 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:39.895476 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:39.895772 (+ 296us) server_negotiation.cc:299] Negotiation successful
0620 18:35:39.895983 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260620 18:35:39.897070 24003 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56096
I20260620 18:35:39.914157 24344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:39.900631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41572 (local address 127.22.37.62:37885)
0620 18:35:39.900854 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:39.900860 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:39.901155 (+ 295us) server_negotiation.cc:407] Connection header received
0620 18:35:39.901277 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:39.901283 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:39.901392 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:39.901485 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:39.902700 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:39.903810 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:39.905045 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:39.905760 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:39.905888 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:39.908008 (+ 2120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:39.908014 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:39.908019 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:39.908056 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:39.911664 (+ 3608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:39.912259 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:39.912264 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:39.912267 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:39.912405 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:39.912726 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:39.912731 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:39.912735 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:39.913090 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:39.913302 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:39.913554 (+ 252us) server_negotiation.cc:299] Negotiation successful
0620 18:35:39.913798 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
W20260620 18:35:39.914804 23799 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41572
I20260620 18:35:39.916985 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 23872
I20260620 18:35:39.930378 24304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:39.918266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56102 (local address 127.22.37.1:44765)
0620 18:35:39.918575 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:39.918583 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:39.918604 (+ 21us) server_negotiation.cc:407] Connection header received
0620 18:35:39.918661 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:39.918665 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:39.918722 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:39.918820 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:39.920104 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:39.920892 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:39.922109 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:39.922509 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:39.922675 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:39.925318 (+ 2643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:39.925329 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:39.925332 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:39.925407 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:39.927913 (+ 2506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:39.928535 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:39.928545 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:39.928548 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:39.928665 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:39.928998 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:39.929003 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:39.929007 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:39.929261 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:39.929513 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:39.929815 (+ 302us) server_negotiation.cc:299] Negotiation successful
0620 18:35:39.930031 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147}
W20260620 18:35:39.930955 24003 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56102
W20260620 18:35:39.931510 22676 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 23872: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:35:39.931699 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 23872
I20260620 18:35:39.954574 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 24012
Jun 20 18:35:39 dist-test-slave-5lgp krb5kdc[23756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980538, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.22.37.2@KRBTEST.COM
I20260620 18:35:39.969956 24302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:39.955309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55774 (local address 127.22.37.2:37905)
0620 18:35:39.955482 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:39.955490 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:39.955624 (+ 134us) server_negotiation.cc:407] Connection header received
0620 18:35:39.955838 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:39.955842 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:39.955906 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:39.956007 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:39.957543 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:39.958380 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:39.959725 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:39.960126 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:39.960259 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:39.964159 (+ 3900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:39.964171 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:39.964186 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:39.964216 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:39.966996 (+ 2780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:39.967628 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:39.967636 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:39.967639 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:39.967760 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:39.968123 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:39.968129 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:39.968132 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:39.968544 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:39.968841 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:39.969177 (+ 336us) server_negotiation.cc:299] Negotiation successful
0620 18:35:39.969442 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260620 18:35:39.970599 24142 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55774
W20260620 18:35:39.971220 22676 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 24012: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:35:39.971396 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 24012
I20260620 18:35:39.990760 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 24151
Jun 20 18:35:39 dist-test-slave-5lgp krb5kdc[23756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980538, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.22.37.3@KRBTEST.COM
I20260620 18:35:40.006434 24300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:39.991517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52090 (local address 127.22.37.3:39949)
0620 18:35:39.991716 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:39.991725 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:39.991846 (+ 121us) server_negotiation.cc:407] Connection header received
0620 18:35:39.992062 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:39.992066 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:39.992133 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:39.992255 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:39.993499 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:39.994261 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:39.995488 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:39.995922 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:39.996081 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:40.000326 (+ 4245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:40.000337 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:40.000353 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:40.000388 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:40.003523 (+ 3135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:40.004242 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:40.004252 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:40.004255 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:40.004416 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:40.004782 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:40.004787 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:40.004791 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:40.005184 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:40.005556 (+ 372us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:40.005848 (+ 292us) server_negotiation.cc:299] Negotiation successful
0620 18:35:40.006107 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260620 18:35:40.007109 24281 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52090
W20260620 18:35:40.007797 22676 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 24151: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:35:40.007948 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 24151
I20260620 18:35:40.028108 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 23769
I20260620 18:35:40.029243 23841 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:35:40.143816 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 23769
2026-06-20T18:35:40Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4536 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:35:40 dist-test-slave-5lgp krb5kdc[24369](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:35:40 dist-test-slave-5lgp krb5kdc[24369](info): set up 2 sockets
Jun 20 18:35:40 dist-test-slave-5lgp krb5kdc[24369](info): commencing operation
krb5kdc: starting...
W20260620 18:35:41.225430 22676 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.002s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/krb5kdc/test-user.keytab.
Jun 20 18:35:41 dist-test-slave-5lgp krb5kdc[24369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980541, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:35:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:35:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.37.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.37.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.37.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab.
Entry for principal kudu/127.22.37.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab.
I20260620 18:35:41.416646 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.62:38817
--webserver_interface=127.22.37.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab
--principal=kudu/127.22.37.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:42907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.37.62:38817
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:35:41.738626 24382 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:35:41.738921 24382 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:35:41.739084 24382 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:35:41.746500 24382 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:35:41.746615 24382 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 18:35:41.746685 24382 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:35:41.746778 24382 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:35:41.746855 24382 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:35:41.746958 24382 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:35:41.756875 24382 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:42907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.37.62:38817
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab
--principal=kudu/127.22.37.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.62:38817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.37.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:35:41.758275 24382 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:35:41.759652 24382 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:35:41.771473 24387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:41.771519 24390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:41.771576 24388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:35:41.772346 24382 server_base.cc:1034] running on GCE node
I20260620 18:35:41.773200 24382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:35:41.774631 24382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:35:41.775871 24382 hybrid_clock.cc:648] HybridClock initialized: now 1781980541775821 us; error 40 us; skew 500 ppm
I20260620 18:35:41.776212 24382 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:35:41 dist-test-slave-5lgp krb5kdc[24369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980541, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:35:41.779228 24382 init.cc:377] Logged in from keytab as kudu/127.22.37.62@KRBTEST.COM (short username kudu)
I20260620 18:35:41.780649 24382 webserver.cc:460] Webserver started at http://127.22.37.62:43563/ using document root <none> and password file <none>
I20260620 18:35:41.781348 24382 fs_manager.cc:359] Metadata directory not provided
I20260620 18:35:41.781525 24382 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:35:41.781836 24382 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:35:41.784762 24382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/minicluster-data/master-0/data/instance:
uuid: "518479382dc0411d87ae9ad184d6250a"
format_stamp: "Formatted at 2026-06-20 18:35:41 on dist-test-slave-5lgp"
server_key: "2a5fe3827b9b23b083af23835a2021a2"
server_key_iv: "7d96f9d0b570721082219aeed2c79086"
server_key_version: "encryptionkey@0"
I20260620 18:35:41.785583 24382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/minicluster-data/master-0/wal/instance:
uuid: "518479382dc0411d87ae9ad184d6250a"
format_stamp: "Formatted at 2026-06-20 18:35:41 on dist-test-slave-5lgp"
server_key: "2a5fe3827b9b23b083af23835a2021a2"
server_key_iv: "7d96f9d0b570721082219aeed2c79086"
server_key_version: "encryptionkey@0"
I20260620 18:35:41.790692 24382 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 18:35:41.794127 24397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:41.795390 24382 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:35:41.795572 24382 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/minicluster-data/master-0/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/minicluster-data/master-0/wal
uuid: "518479382dc0411d87ae9ad184d6250a"
format_stamp: "Formatted at 2026-06-20 18:35:41 on dist-test-slave-5lgp"
server_key: "2a5fe3827b9b23b083af23835a2021a2"
server_key_iv: "7d96f9d0b570721082219aeed2c79086"
server_key_version: "encryptionkey@0"
I20260620 18:35:41.795787 24382 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:35:41.811968 24382 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:35:41.815809 24382 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:35:41.816067 24382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:35:41.837276 24382 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.62:38817
I20260620 18:35:41.837296 24458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.62:38817 every 8 connection(s)
I20260620 18:35:41.838634 24382 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
I20260620 18:35:41.842482 24459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:35:41.844269 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 24382
I20260620 18:35:41.844492 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/minicluster-data/master-0/wal/instance
I20260620 18:35:41.845108 22676 external_mini_cluster.cc:1428] Setting key 0075c9a851b1099aa98509a9700a0b88
I20260620 18:35:41.855623 24459 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 518479382dc0411d87ae9ad184d6250a: Bootstrap starting.
I20260620 18:35:41.859462 24459 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 518479382dc0411d87ae9ad184d6250a: Neither blocks nor log segments found. Creating new log.
I20260620 18:35:41.860486 24459 log.cc:824] T 00000000000000000000000000000000 P 518479382dc0411d87ae9ad184d6250a: Log is configured to *not* fsync() on all Append() calls
Jun 20 18:35:41 dist-test-slave-5lgp krb5kdc[24369](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980541, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM
I20260620 18:35:41.863602 24459 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 518479382dc0411d87ae9ad184d6250a: No bootstrap required, opened a new log
I20260620 18:35:41.869889 24462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:41.847388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39042 (local address 127.22.37.62:38817)
0620 18:35:41.848329 (+ 941us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:41.848346 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:41.848794 (+ 448us) server_negotiation.cc:407] Connection header received
0620 18:35:41.851167 (+ 2373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:41.851214 (+ 47us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:41.851802 (+ 588us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:41.852300 (+ 498us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:41.853746 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:41.855777 (+ 2031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:41.856982 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:41.858134 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:41.858351 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:41.862584 (+ 4233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:41.862592 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:41.862607 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:41.862640 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:41.865711 (+ 3071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:41.866509 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:41.866515 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:41.866520 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:41.866681 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:41.867071 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:41.867080 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:41.867082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:41.867587 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:41.867793 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:41.868394 (+ 601us) server_negotiation.cc:299] Negotiation successful
0620 18:35:41.868800 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":671,"thread_start_us":395,"threads_started":1}
I20260620 18:35:41.872814 24459 raft_consensus.cc:348] T 00000000000000000000000000000000 P 518479382dc0411d87ae9ad184d6250a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "518479382dc0411d87ae9ad184d6250a" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 38817 } }
I20260620 18:35:41.873204 24459 raft_consensus.cc:374] T 00000000000000000000000000000000 P 518479382dc0411d87ae9ad184d6250a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:35:41.873318 24459 raft_consensus.cc:729] T 00000000000000000000000000000000 P 518479382dc0411d87ae9ad184d6250a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 518479382dc0411d87ae9ad184d6250a, State: Initialized, Role: FOLLOWER
I20260620 18:35:41.873905 24459 consensus_queue.cc:260] T 00000000000000000000000000000000 P 518479382dc0411d87ae9ad184d6250a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "518479382dc0411d87ae9ad184d6250a" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 38817 } }
I20260620 18:35:41.874137 24459 raft_consensus.cc:388] T 00000000000000000000000000000000 P 518479382dc0411d87ae9ad184d6250a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:35:41.874289 24459 raft_consensus.cc:482] T 00000000000000000000000000000000 P 518479382dc0411d87ae9ad184d6250a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:35:41.874456 24459 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 518479382dc0411d87ae9ad184d6250a [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:35:41.876575 24459 raft_consensus.cc:504] T 00000000000000000000000000000000 P 518479382dc0411d87ae9ad184d6250a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "518479382dc0411d87ae9ad184d6250a" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 38817 } }
I20260620 18:35:41.877204 24459 leader_election.cc:302] T 00000000000000000000000000000000 P 518479382dc0411d87ae9ad184d6250a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 518479382dc0411d87ae9ad184d6250a; no voters:
I20260620 18:35:41.877777 24459 leader_election.cc:288] T 00000000000000000000000000000000 P 518479382dc0411d87ae9ad184d6250a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:35:41.877931 24464 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 518479382dc0411d87ae9ad184d6250a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:35:41.878650 24464 raft_consensus.cc:686] T 00000000000000000000000000000000 P 518479382dc0411d87ae9ad184d6250a [term 1 LEADER]: Becoming Leader. State: Replica: 518479382dc0411d87ae9ad184d6250a, State: Running, Role: LEADER
I20260620 18:35:41.879215 24464 consensus_queue.cc:237] T 00000000000000000000000000000000 P 518479382dc0411d87ae9ad184d6250a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "518479382dc0411d87ae9ad184d6250a" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 38817 } }
I20260620 18:35:41.879961 24459 sys_catalog.cc:564] T 00000000000000000000000000000000 P 518479382dc0411d87ae9ad184d6250a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:35:41.882179 24465 sys_catalog.cc:455] T 00000000000000000000000000000000 P 518479382dc0411d87ae9ad184d6250a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "518479382dc0411d87ae9ad184d6250a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "518479382dc0411d87ae9ad184d6250a" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 38817 } } }
I20260620 18:35:41.882185 24466 sys_catalog.cc:455] T 00000000000000000000000000000000 P 518479382dc0411d87ae9ad184d6250a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 518479382dc0411d87ae9ad184d6250a. Latest consensus state: current_term: 1 leader_uuid: "518479382dc0411d87ae9ad184d6250a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "518479382dc0411d87ae9ad184d6250a" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 38817 } } }
I20260620 18:35:41.882609 24466 sys_catalog.cc:458] T 00000000000000000000000000000000 P 518479382dc0411d87ae9ad184d6250a [sys.catalog]: This master's current role is: LEADER
I20260620 18:35:41.882608 24465 sys_catalog.cc:458] T 00000000000000000000000000000000 P 518479382dc0411d87ae9ad184d6250a [sys.catalog]: This master's current role is: LEADER
I20260620 18:35:41.883556 24470 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:35:41.890880 24470 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:35:41.899809 24470 catalog_manager.cc:1347] Generated new cluster ID: a345ae36835c454b95de455b23d613b6
I20260620 18:35:41.899914 24470 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:35:41.925307 24470 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:35:41.927376 24470 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:35:41.938020 24470 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 518479382dc0411d87ae9ad184d6250a: Generated new TSK 0
I20260620 18:35:41.939109 24470 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.37.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.37.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.37.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.1.keytab.
Entry for principal kudu/127.22.37.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.1.keytab.
I20260620 18:35:41.989303 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.37.1:0
--local_ip_for_outbound_sockets=127.22.37.1
--webserver_interface=127.22.37.1
--webserver_port=0
--tserver_master_addrs=127.22.37.62:38817
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.1.keytab
--principal=kudu/127.22.37.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:42907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:35:42.308029 24485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:35:42.308333 24485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:35:42.308499 24485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:35:42.315709 24485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:35:42.315825 24485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:35:42.315977 24485 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 18:35:42.316059 24485 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 18:35:42.316184 24485 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.37.1
I20260620 18:35:42.326191 24485 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:42907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.1.keytab
--principal=kudu/127.22.37.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.37.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.37.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.37.62:38817
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.37.1
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:35:42.327574 24485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:35:42.328910 24485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:35:42.342459 24490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:42.342422 24491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:42.342446 24493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:35:42.343263 24485 server_base.cc:1034] running on GCE node
I20260620 18:35:42.343878 24485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:35:42.344771 24485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:35:42.345993 24485 hybrid_clock.cc:648] HybridClock initialized: now 1781980542345943 us; error 46 us; skew 500 ppm
I20260620 18:35:42.346294 24485 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:35:42 dist-test-slave-5lgp krb5kdc[24369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980542, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:35:42.349325 24485 init.cc:377] Logged in from keytab as kudu/127.22.37.1@KRBTEST.COM (short username kudu)
I20260620 18:35:42.350739 24485 webserver.cc:460] Webserver started at http://127.22.37.1:38949/ using document root <none> and password file <none>
I20260620 18:35:42.351420 24485 fs_manager.cc:359] Metadata directory not provided
I20260620 18:35:42.351557 24485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:35:42.351857 24485 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:35:42.354728 24485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/minicluster-data/ts-0/data/instance:
uuid: "f5ec1f95ec4544deb9e4ba19316d3e02"
format_stamp: "Formatted at 2026-06-20 18:35:42 on dist-test-slave-5lgp"
server_key: "a3ccbe76662441f69825573399ec4ae3"
server_key_iv: "3714794365b6369ab8e0941e0d67b7da"
server_key_version: "encryptionkey@0"
I20260620 18:35:42.355458 24485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/minicluster-data/ts-0/wal/instance:
uuid: "f5ec1f95ec4544deb9e4ba19316d3e02"
format_stamp: "Formatted at 2026-06-20 18:35:42 on dist-test-slave-5lgp"
server_key: "a3ccbe76662441f69825573399ec4ae3"
server_key_iv: "3714794365b6369ab8e0941e0d67b7da"
server_key_version: "encryptionkey@0"
I20260620 18:35:42.360369 24485 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 18:35:42.363739 24500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:42.365041 24485 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 18:35:42.365203 24485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/minicluster-data/ts-0/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/minicluster-data/ts-0/wal
uuid: "f5ec1f95ec4544deb9e4ba19316d3e02"
format_stamp: "Formatted at 2026-06-20 18:35:42 on dist-test-slave-5lgp"
server_key: "a3ccbe76662441f69825573399ec4ae3"
server_key_iv: "3714794365b6369ab8e0941e0d67b7da"
server_key_version: "encryptionkey@0"
I20260620 18:35:42.365415 24485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:35:42.377548 24485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:35:42.382149 24485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:35:42.382439 24485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:35:42.385178 24485 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:35:42.385336 24485 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:42.385497 24485 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:35:42.385630 24485 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 18:35:42 dist-test-slave-5lgp krb5kdc[24369](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980542, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.1@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM
I20260620 18:35:42.407300 24509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:42.385813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.1:49669 (local address 127.22.37.62:38817)
0620 18:35:42.386633 (+ 820us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:42.386641 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:42.387475 (+ 834us) server_negotiation.cc:407] Connection header received
0620 18:35:42.389656 (+ 2181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:42.389662 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:42.389753 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:42.389868 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:42.391994 (+ 2126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:42.392692 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:42.394924 (+ 2232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:42.395537 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:42.395688 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:42.401121 (+ 5433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:42.401126 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:42.401129 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:42.401162 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:42.403933 (+ 2771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:42.404954 (+ 1021us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:42.404959 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:42.404968 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:42.405085 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:42.405561 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:42.405567 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:42.405575 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:42.405843 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:42.406000 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:42.406688 (+ 688us) server_negotiation.cc:299] Negotiation successful
0620 18:35:42.406946 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":701,"thread_start_us":587,"threads_started":1}
I20260620 18:35:42.408298 24510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:42.386505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:38817 (local address 127.22.37.1:49669)
0620 18:35:42.387313 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:42.387349 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:42.389340 (+ 1991us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:42.390166 (+ 826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:42.390181 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:42.391043 (+ 862us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:42.391816 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:42.391834 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:42.392835 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:42.392841 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:42.394744 (+ 1903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:42.394758 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:42.395809 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:42.395821 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:42.396205 (+ 384us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:42.397252 (+ 1047us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:35:42.397298 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:42.400885 (+ 3587us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:35:42.404079 (+ 3194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:42.404094 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:42.404115 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:42.404761 (+ 646us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:42.405195 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:42.405201 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:42.405204 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:42.405449 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:42.405979 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:35:42.405987 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:35:42.406334 (+ 347us) client_negotiation.cc:769] Sending connection context
0620 18:35:42.406602 (+ 268us) client_negotiation.cc:240] Negotiation successful
0620 18:35:42.407144 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":359,"threads_started":1}
I20260620 18:35:42.430640 24509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:42.414631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.1:45319 (local address 127.22.37.62:38817)
0620 18:35:42.414862 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:42.414870 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:42.415464 (+ 594us) server_negotiation.cc:407] Connection header received
0620 18:35:42.415614 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:42.415623 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:42.415733 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:42.415864 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:42.416727 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:42.418062 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:42.420846 (+ 2784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:42.421254 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:42.421442 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:42.424234 (+ 2792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:42.424243 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:42.424250 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:42.424294 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:42.427508 (+ 3214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:42.428355 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:42.428363 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:42.428367 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:42.428537 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:42.428960 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:42.428967 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:42.428972 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:42.429331 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:42.429585 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:42.430176 (+ 591us) server_negotiation.cc:299] Negotiation successful
0620 18:35:42.430406 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 18:35:42.431202 24563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:42.414719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:38817 (local address 127.22.37.1:45319)
0620 18:35:42.415351 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:42.415378 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:42.415496 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:42.415875 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:42.415878 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:42.416299 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:42.416596 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:42.416604 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:42.418267 (+ 1663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:42.418273 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:42.420683 (+ 2410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:42.420703 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:42.421503 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:42.421513 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:42.421805 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:42.422569 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:35:42.422595 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:42.424012 (+ 1417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:35:42.427631 (+ 3619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:42.427638 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:42.427644 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:42.428219 (+ 575us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:42.428641 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:42.428645 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:42.428649 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:42.428789 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:42.429555 (+ 766us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:35:42.429563 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:35:42.429834 (+ 271us) client_negotiation.cc:769] Sending connection context
0620 18:35:42.430069 (+ 235us) client_negotiation.cc:240] Negotiation successful
0620 18:35:42.430548 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":356,"threads_started":1}
I20260620 18:35:42.441814 24485 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.1:39519
I20260620 18:35:42.441845 24620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.1:39519 every 8 connection(s)
I20260620 18:35:42.443142 24485 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/minicluster-data/ts-0/data/info.pb
I20260620 18:35:42.445163 24621 heartbeater.cc:347] Connected to a master server at 127.22.37.62:38817
I20260620 18:35:42.445425 24621 heartbeater.cc:464] Registering TS with master...
I20260620 18:35:42.446045 24621 heartbeater.cc:511] Master 127.22.37.62:38817 requested a full tablet report, sending...
I20260620 18:35:42.447883 24414 ts_manager.cc:194] Registered new tserver with Master: f5ec1f95ec4544deb9e4ba19316d3e02 (127.22.37.1:39519)
I20260620 18:35:42.447969 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 24485
I20260620 18:35:42.448160 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/minicluster-data/ts-0/wal/instance
I20260620 18:35:42.448796 22676 external_mini_cluster.cc:1428] Setting key 89e6945c4c0e6bdcb20f7d19b3c660c9
I20260620 18:35:42.449491 24414 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.37.1@KRBTEST.COM'} at 127.22.37.1:49669
WARNING: no policy specified for kudu/127.22.37.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.37.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.37.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.2.keytab.
Entry for principal kudu/127.22.37.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.2.keytab.
I20260620 18:35:42.487059 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.37.2:0
--local_ip_for_outbound_sockets=127.22.37.2
--webserver_interface=127.22.37.2
--webserver_port=0
--tserver_master_addrs=127.22.37.62:38817
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.2.keytab
--principal=kudu/127.22.37.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:42907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:35:42.803354 24625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:35:42.803679 24625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:35:42.803851 24625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:35:42.811088 24625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:35:42.811232 24625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:35:42.811373 24625 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 18:35:42.811455 24625 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 18:35:42.811582 24625 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.37.2
I20260620 18:35:42.821288 24625 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:42907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.2.keytab
--principal=kudu/127.22.37.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.37.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.37.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.37.62:38817
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.37.2
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:35:42.822722 24625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:35:42.824051 24625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:35:42.837174 24633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:42.837152 24631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:42.842172 24630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:35:42.843025 24625 server_base.cc:1034] running on GCE node
I20260620 18:35:42.843685 24625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:35:42.844635 24625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:35:42.845845 24625 hybrid_clock.cc:648] HybridClock initialized: now 1781980542845799 us; error 43 us; skew 500 ppm
I20260620 18:35:42.846158 24625 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:35:42 dist-test-slave-5lgp krb5kdc[24369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980542, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:35:42.849282 24625 init.cc:377] Logged in from keytab as kudu/127.22.37.2@KRBTEST.COM (short username kudu)
I20260620 18:35:42.850890 24625 webserver.cc:460] Webserver started at http://127.22.37.2:35633/ using document root <none> and password file <none>
I20260620 18:35:42.851564 24625 fs_manager.cc:359] Metadata directory not provided
I20260620 18:35:42.851684 24625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:35:42.851998 24625 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:35:42.854949 24625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/minicluster-data/ts-1/data/instance:
uuid: "819af79e262244839ea1ff6ed5aec230"
format_stamp: "Formatted at 2026-06-20 18:35:42 on dist-test-slave-5lgp"
server_key: "0549d09a3e19d1257c08a0a1deb22a57"
server_key_iv: "504437925b50bcd53e33e49cd2ef0b3f"
server_key_version: "encryptionkey@0"
I20260620 18:35:42.855701 24625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/minicluster-data/ts-1/wal/instance:
uuid: "819af79e262244839ea1ff6ed5aec230"
format_stamp: "Formatted at 2026-06-20 18:35:42 on dist-test-slave-5lgp"
server_key: "0549d09a3e19d1257c08a0a1deb22a57"
server_key_iv: "504437925b50bcd53e33e49cd2ef0b3f"
server_key_version: "encryptionkey@0"
I20260620 18:35:42.860915 24625 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260620 18:35:42.864533 24640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:42.865957 24625 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260620 18:35:42.866124 24625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/minicluster-data/ts-1/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/minicluster-data/ts-1/wal
uuid: "819af79e262244839ea1ff6ed5aec230"
format_stamp: "Formatted at 2026-06-20 18:35:42 on dist-test-slave-5lgp"
server_key: "0549d09a3e19d1257c08a0a1deb22a57"
server_key_iv: "504437925b50bcd53e33e49cd2ef0b3f"
server_key_version: "encryptionkey@0"
I20260620 18:35:42.866338 24625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:35:42.895076 24625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:35:42.899012 24625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:35:42.899303 24625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:35:42.901971 24625 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:35:42.902107 24625 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:42.902233 24625 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:35:42.902330 24625 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 18:35:42 dist-test-slave-5lgp krb5kdc[24369](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980542, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.2@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM
I20260620 18:35:42.928191 24649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:42.903159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:38817 (local address 127.22.37.2:51065)
0620 18:35:42.904107 (+ 948us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:42.904162 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:42.906167 (+ 2005us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:42.907111 (+ 944us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:42.907119 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:42.907861 (+ 742us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:42.908765 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:42.908788 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:42.910250 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:42.910259 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:42.913252 (+ 2993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:42.913273 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:42.914385 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:42.914396 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:42.914786 (+ 390us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:42.915597 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:35:42.915635 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:42.919288 (+ 3653us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:35:42.922528 (+ 3240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:42.922538 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:42.922560 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:42.923224 (+ 664us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:42.923650 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:42.923660 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:42.923663 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:42.923939 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:42.924645 (+ 706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:35:42.924658 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:35:42.925240 (+ 582us) client_negotiation.cc:769] Sending connection context
0620 18:35:42.925671 (+ 431us) client_negotiation.cc:240] Negotiation successful
0620 18:35:42.926403 (+ 732us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"thread_start_us":400,"threads_started":1}
I20260620 18:35:42.928325 24509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:42.902503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.2:51065 (local address 127.22.37.62:38817)
0620 18:35:42.902708 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:42.902715 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:42.904292 (+ 1577us) server_negotiation.cc:407] Connection header received
0620 18:35:42.906547 (+ 2255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:42.906557 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:42.906713 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:42.906841 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:42.908990 (+ 2149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:42.910067 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:42.913473 (+ 3406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:42.914079 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:42.914246 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:42.919532 (+ 5286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:42.919537 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:42.919541 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:42.919574 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:42.922303 (+ 2729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:42.923350 (+ 1047us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:42.923355 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:42.923359 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:42.923525 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:42.924092 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:42.924099 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:42.924105 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:42.924409 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:42.924654 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:42.927540 (+ 2886us) server_negotiation.cc:299] Negotiation successful
0620 18:35:42.927833 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 18:35:42.948259 24509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:42.933806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.2:33069 (local address 127.22.37.62:38817)
0620 18:35:42.933948 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:42.933952 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:42.934538 (+ 586us) server_negotiation.cc:407] Connection header received
0620 18:35:42.934709 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:42.934715 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:42.934783 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:42.934888 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:42.935921 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:42.936835 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:42.938711 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:42.939626 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:42.939790 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:42.941502 (+ 1712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:42.941512 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:42.941520 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:42.941562 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:42.945511 (+ 3949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:42.946378 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:42.946389 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:42.946391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:42.946514 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:42.946850 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:42.946857 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:42.946861 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:42.947165 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:42.947312 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:42.947686 (+ 374us) server_negotiation.cc:299] Negotiation successful
0620 18:35:42.947897 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 18:35:42.948349 24711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:42.933847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:38817 (local address 127.22.37.2:33069)
0620 18:35:42.934430 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:42.934461 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:42.934600 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:42.934904 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:42.934908 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:42.935505 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:42.935780 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:42.935788 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:42.937017 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:42.937025 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:42.938574 (+ 1549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:42.938585 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:42.939803 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:42.939807 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:42.939919 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:42.940596 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:35:42.940612 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:42.941284 (+ 672us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:35:42.945639 (+ 4355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:42.945650 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:42.945654 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:42.946198 (+ 544us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:42.946626 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:42.946632 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:42.946636 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:42.946752 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:42.947276 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:35:42.947281 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:35:42.947463 (+ 182us) client_negotiation.cc:769] Sending connection context
0620 18:35:42.947601 (+ 138us) client_negotiation.cc:240] Negotiation successful
0620 18:35:42.947904 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":345,"threads_started":1}
I20260620 18:35:42.961788 24625 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.2:33193
I20260620 18:35:42.961795 24759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.2:33193 every 8 connection(s)
I20260620 18:35:42.963173 24625 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/minicluster-data/ts-1/data/info.pb
I20260620 18:35:42.964519 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 24625
I20260620 18:35:42.964735 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/minicluster-data/ts-1/wal/instance
I20260620 18:35:42.965340 22676 external_mini_cluster.cc:1428] Setting key 2f63fab01433fb0f56228a8bf498007d
I20260620 18:35:42.965349 24760 heartbeater.cc:347] Connected to a master server at 127.22.37.62:38817
I20260620 18:35:42.965615 24760 heartbeater.cc:464] Registering TS with master...
I20260620 18:35:42.966459 24760 heartbeater.cc:511] Master 127.22.37.62:38817 requested a full tablet report, sending...
I20260620 18:35:42.967914 24413 ts_manager.cc:194] Registered new tserver with Master: 819af79e262244839ea1ff6ed5aec230 (127.22.37.2:33193)
I20260620 18:35:42.968827 24413 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.37.2@KRBTEST.COM'} at 127.22.37.2:51065
WARNING: no policy specified for kudu/127.22.37.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.37.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.37.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.3.keytab.
Entry for principal kudu/127.22.37.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.3.keytab.
I20260620 18:35:43.005168 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.37.3:0
--local_ip_for_outbound_sockets=127.22.37.3
--webserver_interface=127.22.37.3
--webserver_port=0
--tserver_master_addrs=127.22.37.62:38817
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.3.keytab
--principal=kudu/127.22.37.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:42907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:35:43.322674 24764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:35:43.322993 24764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:35:43.323166 24764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:35:43.330397 24764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:35:43.330542 24764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:35:43.330675 24764 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 18:35:43.330760 24764 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 18:35:43.330883 24764 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.37.3
I20260620 18:35:43.340842 24764 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:42907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.3.keytab
--principal=kudu/127.22.37.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.37.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.37.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.37.62:38817
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.37.3
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:35:43.342213 24764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:35:43.343487 24764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:35:43.356499 24769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:43.356493 24770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:43.356427 24772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:35:43.357439 24764 server_base.cc:1034] running on GCE node
I20260620 18:35:43.358047 24764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:35:43.358978 24764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:35:43.360174 24764 hybrid_clock.cc:648] HybridClock initialized: now 1781980543360123 us; error 40 us; skew 500 ppm
I20260620 18:35:43.360600 24764 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:35:43 dist-test-slave-5lgp krb5kdc[24369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980543, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:35:43.363765 24764 init.cc:377] Logged in from keytab as kudu/127.22.37.3@KRBTEST.COM (short username kudu)
I20260620 18:35:43.365214 24764 webserver.cc:460] Webserver started at http://127.22.37.3:40385/ using document root <none> and password file <none>
I20260620 18:35:43.365939 24764 fs_manager.cc:359] Metadata directory not provided
I20260620 18:35:43.366079 24764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:35:43.366382 24764 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:35:43.369436 24764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/minicluster-data/ts-2/data/instance:
uuid: "a6facfe4b14c490e8cc97f3d40b6d4df"
format_stamp: "Formatted at 2026-06-20 18:35:43 on dist-test-slave-5lgp"
server_key: "c4f5db6f36c7a63a3163a3bee083abf1"
server_key_iv: "32926c68a9d73db29c7cb74667c13a7c"
server_key_version: "encryptionkey@0"
I20260620 18:35:43.370249 24764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/minicluster-data/ts-2/wal/instance:
uuid: "a6facfe4b14c490e8cc97f3d40b6d4df"
format_stamp: "Formatted at 2026-06-20 18:35:43 on dist-test-slave-5lgp"
server_key: "c4f5db6f36c7a63a3163a3bee083abf1"
server_key_iv: "32926c68a9d73db29c7cb74667c13a7c"
server_key_version: "encryptionkey@0"
I20260620 18:35:43.375236 24764 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:35:43.378516 24779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:43.379799 24764 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:35:43.379954 24764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/minicluster-data/ts-2/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/minicluster-data/ts-2/wal
uuid: "a6facfe4b14c490e8cc97f3d40b6d4df"
format_stamp: "Formatted at 2026-06-20 18:35:43 on dist-test-slave-5lgp"
server_key: "c4f5db6f36c7a63a3163a3bee083abf1"
server_key_iv: "32926c68a9d73db29c7cb74667c13a7c"
server_key_version: "encryptionkey@0"
I20260620 18:35:43.380136 24764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:35:43.419471 24764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:35:43.423604 24764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:35:43.423904 24764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:35:43.426677 24764 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:35:43.426829 24764 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:43.426954 24764 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:35:43.427057 24764 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 18:35:43 dist-test-slave-5lgp krb5kdc[24369](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980543, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.3@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM
I20260620 18:35:43.449935 24509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:43.427233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.3:50497 (local address 127.22.37.62:38817)
0620 18:35:43.427432 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:43.427439 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:43.428763 (+ 1324us) server_negotiation.cc:407] Connection header received
0620 18:35:43.431055 (+ 2292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:43.431064 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:43.431168 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:43.431292 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:43.433599 (+ 2307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:43.434942 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:43.436549 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:43.437327 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:43.437497 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:43.442127 (+ 4630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:43.442139 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:43.442148 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:43.442205 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:43.446673 (+ 4468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:43.447566 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:43.447576 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:43.447579 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:43.447705 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:43.448136 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:43.448140 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:43.448148 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:43.448414 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:43.448588 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:43.449300 (+ 712us) server_negotiation.cc:299] Negotiation successful
0620 18:35:43.449578 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 18:35:43.450829 24788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:43.427748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:38817 (local address 127.22.37.3:50497)
0620 18:35:43.428579 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:43.428629 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:43.430779 (+ 2150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:43.431587 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:43.431598 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:43.432442 (+ 844us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:43.433341 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:43.433401 (+ 60us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:43.435129 (+ 1728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:43.435137 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:43.436411 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:43.436422 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:43.437652 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:43.437663 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:43.438009 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:43.438736 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:35:43.438759 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:43.441851 (+ 3092us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:35:43.446864 (+ 5013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:43.446874 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:43.446889 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:43.447417 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:43.447812 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:43.447818 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:43.447822 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:43.448019 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:43.448556 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:35:43.448566 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:35:43.448913 (+ 347us) client_negotiation.cc:769] Sending connection context
0620 18:35:43.449168 (+ 255us) client_negotiation.cc:240] Negotiation successful
0620 18:35:43.449710 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":362,"threads_started":1}
I20260620 18:35:43.452261 24621 heartbeater.cc:503] Master 127.22.37.62:38817 was elected leader, sending a full tablet report...
I20260620 18:35:43.476389 24839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:43.457768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:38817 (local address 127.22.37.3:42305)
0620 18:35:43.461738 (+ 3970us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:43.461765 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:43.461979 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:43.462386 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:43.462393 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:43.463195 (+ 802us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:43.463724 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:43.463742 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:43.464897 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:43.464906 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:43.466294 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:43.466306 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:43.467449 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:43.467453 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:43.467658 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:43.468459 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:35:43.468488 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:43.469828 (+ 1340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:35:43.473871 (+ 4043us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:43.473876 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:43.473880 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:43.474285 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:43.474665 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:43.474670 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:43.474673 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:43.474875 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:43.475586 (+ 711us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:35:43.475590 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:35:43.475742 (+ 152us) client_negotiation.cc:769] Sending connection context
0620 18:35:43.475846 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 18:35:43.476078 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3772,"thread_start_us":3578,"threads_started":1}
I20260620 18:35:43.476792 24509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:43.458283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.3:42305 (local address 127.22.37.62:38817)
0620 18:35:43.458453 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:43.458459 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:43.461856 (+ 3397us) server_negotiation.cc:407] Connection header received
0620 18:35:43.462127 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:43.462134 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:43.462225 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:43.462328 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:43.463893 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:43.464754 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:43.466501 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:43.467083 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:43.467194 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:43.470087 (+ 2893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:43.470096 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:43.470101 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:43.470137 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:43.473748 (+ 3611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:43.474406 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:43.474412 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:43.474415 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:43.474546 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:43.475018 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:43.475026 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:43.475030 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:43.475465 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:43.475649 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:43.476016 (+ 367us) server_negotiation.cc:299] Negotiation successful
0620 18:35:43.476315 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 18:35:43.485167 24764 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.3:35601
I20260620 18:35:43.485204 24898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.3:35601 every 8 connection(s)
I20260620 18:35:43.486491 24764 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/minicluster-data/ts-2/data/info.pb
I20260620 18:35:43.488483 24899 heartbeater.cc:347] Connected to a master server at 127.22.37.62:38817
I20260620 18:35:43.488710 24899 heartbeater.cc:464] Registering TS with master...
I20260620 18:35:43.489332 24899 heartbeater.cc:511] Master 127.22.37.62:38817 requested a full tablet report, sending...
I20260620 18:35:43.490687 24413 ts_manager.cc:194] Registered new tserver with Master: a6facfe4b14c490e8cc97f3d40b6d4df (127.22.37.3:35601)
I20260620 18:35:43.491454 24413 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.37.3@KRBTEST.COM'} at 127.22.37.3:50497
I20260620 18:35:43.493361 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 24764
I20260620 18:35:43.493583 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/minicluster-data/ts-2/wal/instance
I20260620 18:35:43.494015 22676 external_mini_cluster.cc:1428] Setting key eedff1451ced8c101b498994caa981db
I20260620 18:35:43.496693 22676 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 18:35:43 dist-test-slave-5lgp krb5kdc[24369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 18:35:43 dist-test-slave-5lgp krb5kdc[24369](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM
I20260620 18:35:43.541471 24509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:43.527715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39058 (local address 127.22.37.62:38817)
0620 18:35:43.527940 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:43.527946 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:43.528093 (+ 147us) server_negotiation.cc:407] Connection header received
0620 18:35:43.528244 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:43.528248 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:43.528313 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:43.528399 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:43.529855 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:43.530640 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:43.531853 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:43.532238 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:43.532336 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:43.536365 (+ 4029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:43.536370 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:43.536373 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:43.536410 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:43.539150 (+ 2740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:43.539778 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:43.539781 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:43.539784 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:43.539901 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:43.540223 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:43.540226 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:43.540228 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:43.540494 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:43.540673 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:43.540915 (+ 242us) server_negotiation.cc:299] Negotiation successful
0620 18:35:43.541078 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 18:35:43.545468 24413 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39058:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:35:43.552203 24413 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:35:43.574198 24913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:43.563905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54838 (local address 127.22.37.1:39519)
0620 18:35:43.564464 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:43.564473 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:43.565267 (+ 794us) server_negotiation.cc:407] Connection header received
0620 18:35:43.565622 (+ 355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:43.565633 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:43.565887 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:43.566049 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:35:43.567022 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:43.568559 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:43.571361 (+ 2802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:43.572758 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:43.572850 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:43.573049 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:43.573707 (+ 658us) server_negotiation.cc:299] Negotiation successful
0620 18:35:43.573886 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":296,"threads_started":1}
I20260620 18:35:43.574746 24912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:43.563920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.2:33193 (local address 127.0.0.1:57084)
0620 18:35:43.565084 (+ 1164us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:43.565162 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:43.565333 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:43.566488 (+ 1155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:43.566492 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:43.566511 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:35:43.566977 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:43.566992 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:43.569701 (+ 2709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:43.569708 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:43.571869 (+ 2161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:43.571882 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:43.573873 (+ 1991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:43.573879 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:43.574055 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:43.574087 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 18:35:43.574154 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 18:35:43.574356 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":941,"mutex_wait_us":1,"thread_start_us":810,"threads_started":1}
I20260620 18:35:43.575160 24914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:43.563730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.1:39519 (local address 127.0.0.1:54838)
0620 18:35:43.565084 (+ 1354us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:43.565162 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:43.565344 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:43.566295 (+ 951us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:43.566302 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:43.566334 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:35:43.566783 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:43.566803 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:43.568731 (+ 1928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:43.568736 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:43.571198 (+ 2462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:43.571212 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:43.573227 (+ 2015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:43.573233 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:43.573437 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:43.573490 (+ 53us) client_negotiation.cc:769] Sending connection context
0620 18:35:43.574696 (+ 1206us) client_negotiation.cc:240] Negotiation successful
0620 18:35:43.574843 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1072,"thread_start_us":1659,"threads_started":1}
I20260620 18:35:43.576109 24915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:43.564561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57084 (local address 127.22.37.2:33193)
0620 18:35:43.565928 (+ 1367us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:43.565936 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:43.565959 (+ 23us) server_negotiation.cc:407] Connection header received
0620 18:35:43.566060 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:43.566067 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:43.566291 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:43.567166 (+ 875us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:35:43.567337 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:43.569486 (+ 2149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:43.572340 (+ 2854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:43.573672 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:43.575329 (+ 1657us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:43.575539 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:43.575606 (+ 67us) server_negotiation.cc:299] Negotiation successful
0620 18:35:43.575797 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1223,"thread_start_us":1103,"threads_started":1}
I20260620 18:35:43.579382 24551 tablet_service.cc:1467] Processing CreateTablet for tablet 0266d73c65cb497e9dc91469a90ae127 (DEFAULT_TABLE table=test-table [id=9228f399216e4435b48f1b9e8df886b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:35:43.579555 24690 tablet_service.cc:1467] Processing CreateTablet for tablet 0266d73c65cb497e9dc91469a90ae127 (DEFAULT_TABLE table=test-table [id=9228f399216e4435b48f1b9e8df886b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:35:43.580664 24690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0266d73c65cb497e9dc91469a90ae127. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:35:43.580735 24551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0266d73c65cb497e9dc91469a90ae127. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:35:43.584136 24912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:43.565806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.3:35601 (local address 127.0.0.1:60390)
0620 18:35:43.574959 (+ 9153us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:43.574978 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:43.575077 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:43.577435 (+ 2358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:43.577440 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:43.577466 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:35:43.577626 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:43.577639 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:43.579523 (+ 1884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:43.579528 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:43.581501 (+ 1973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:43.581515 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:43.583381 (+ 1866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:43.583390 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:43.583590 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:43.583627 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 18:35:43.583715 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 18:35:43.583851 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9057,"thread_start_us":10118,"threads_started":1}
I20260620 18:35:43.584339 24917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:43.566390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60390 (local address 127.22.37.3:35601)
0620 18:35:43.576828 (+ 10438us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:43.576836 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:43.576873 (+ 37us) server_negotiation.cc:407] Connection header received
0620 18:35:43.576965 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:43.576974 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:43.577218 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:43.577440 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:35:43.577768 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:43.579370 (+ 1602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:43.581694 (+ 2324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:43.583200 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:43.583329 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:43.583561 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:43.583811 (+ 250us) server_negotiation.cc:299] Negotiation successful
0620 18:35:43.583979 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10266,"thread_start_us":10070,"threads_started":1}
I20260620 18:35:43.588376 24833 tablet_service.cc:1467] Processing CreateTablet for tablet 0266d73c65cb497e9dc91469a90ae127 (DEFAULT_TABLE table=test-table [id=9228f399216e4435b48f1b9e8df886b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:35:43.589928 24833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0266d73c65cb497e9dc91469a90ae127. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:35:43.593191 24918 tablet_bootstrap.cc:492] T 0266d73c65cb497e9dc91469a90ae127 P 819af79e262244839ea1ff6ed5aec230: Bootstrap starting.
I20260620 18:35:43.595745 24919 tablet_bootstrap.cc:492] T 0266d73c65cb497e9dc91469a90ae127 P f5ec1f95ec4544deb9e4ba19316d3e02: Bootstrap starting.
I20260620 18:35:43.597183 24918 tablet_bootstrap.cc:654] T 0266d73c65cb497e9dc91469a90ae127 P 819af79e262244839ea1ff6ed5aec230: Neither blocks nor log segments found. Creating new log.
I20260620 18:35:43.598564 24918 log.cc:824] T 0266d73c65cb497e9dc91469a90ae127 P 819af79e262244839ea1ff6ed5aec230: Log is configured to *not* fsync() on all Append() calls
I20260620 18:35:43.599421 24919 tablet_bootstrap.cc:654] T 0266d73c65cb497e9dc91469a90ae127 P f5ec1f95ec4544deb9e4ba19316d3e02: Neither blocks nor log segments found. Creating new log.
I20260620 18:35:43.600560 24919 log.cc:824] T 0266d73c65cb497e9dc91469a90ae127 P f5ec1f95ec4544deb9e4ba19316d3e02: Log is configured to *not* fsync() on all Append() calls
I20260620 18:35:43.601859 24918 tablet_bootstrap.cc:492] T 0266d73c65cb497e9dc91469a90ae127 P 819af79e262244839ea1ff6ed5aec230: No bootstrap required, opened a new log
I20260620 18:35:43.602159 24918 ts_tablet_manager.cc:1397] T 0266d73c65cb497e9dc91469a90ae127 P 819af79e262244839ea1ff6ed5aec230: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260620 18:35:43.603525 24919 tablet_bootstrap.cc:492] T 0266d73c65cb497e9dc91469a90ae127 P f5ec1f95ec4544deb9e4ba19316d3e02: No bootstrap required, opened a new log
I20260620 18:35:43.603852 24919 ts_tablet_manager.cc:1397] T 0266d73c65cb497e9dc91469a90ae127 P f5ec1f95ec4544deb9e4ba19316d3e02: Time spent bootstrapping tablet: real 0.008s user 0.008s sys 0.000s
I20260620 18:35:43.606096 24922 tablet_bootstrap.cc:492] T 0266d73c65cb497e9dc91469a90ae127 P a6facfe4b14c490e8cc97f3d40b6d4df: Bootstrap starting.
I20260620 18:35:43.608868 24922 tablet_bootstrap.cc:654] T 0266d73c65cb497e9dc91469a90ae127 P a6facfe4b14c490e8cc97f3d40b6d4df: Neither blocks nor log segments found. Creating new log.
I20260620 18:35:43.609850 24922 log.cc:824] T 0266d73c65cb497e9dc91469a90ae127 P a6facfe4b14c490e8cc97f3d40b6d4df: Log is configured to *not* fsync() on all Append() calls
I20260620 18:35:43.612376 24922 tablet_bootstrap.cc:492] T 0266d73c65cb497e9dc91469a90ae127 P a6facfe4b14c490e8cc97f3d40b6d4df: No bootstrap required, opened a new log
I20260620 18:35:43.612612 24922 ts_tablet_manager.cc:1397] T 0266d73c65cb497e9dc91469a90ae127 P a6facfe4b14c490e8cc97f3d40b6d4df: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260620 18:35:43.612782 24918 raft_consensus.cc:348] T 0266d73c65cb497e9dc91469a90ae127 P 819af79e262244839ea1ff6ed5aec230 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ec1f95ec4544deb9e4ba19316d3e02" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 39519 } } peers { permanent_uuid: "819af79e262244839ea1ff6ed5aec230" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 33193 } } peers { permanent_uuid: "a6facfe4b14c490e8cc97f3d40b6d4df" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35601 } }
I20260620 18:35:43.613231 24918 raft_consensus.cc:374] T 0266d73c65cb497e9dc91469a90ae127 P 819af79e262244839ea1ff6ed5aec230 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:35:43.613390 24918 raft_consensus.cc:729] T 0266d73c65cb497e9dc91469a90ae127 P 819af79e262244839ea1ff6ed5aec230 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819af79e262244839ea1ff6ed5aec230, State: Initialized, Role: FOLLOWER
I20260620 18:35:43.614015 24918 consensus_queue.cc:260] T 0266d73c65cb497e9dc91469a90ae127 P 819af79e262244839ea1ff6ed5aec230 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ec1f95ec4544deb9e4ba19316d3e02" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 39519 } } peers { permanent_uuid: "819af79e262244839ea1ff6ed5aec230" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 33193 } } peers { permanent_uuid: "a6facfe4b14c490e8cc97f3d40b6d4df" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35601 } }
I20260620 18:35:43.614286 24919 raft_consensus.cc:348] T 0266d73c65cb497e9dc91469a90ae127 P f5ec1f95ec4544deb9e4ba19316d3e02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ec1f95ec4544deb9e4ba19316d3e02" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 39519 } } peers { permanent_uuid: "819af79e262244839ea1ff6ed5aec230" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 33193 } } peers { permanent_uuid: "a6facfe4b14c490e8cc97f3d40b6d4df" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35601 } }
I20260620 18:35:43.614800 24919 raft_consensus.cc:374] T 0266d73c65cb497e9dc91469a90ae127 P f5ec1f95ec4544deb9e4ba19316d3e02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:35:43.614935 24919 raft_consensus.cc:729] T 0266d73c65cb497e9dc91469a90ae127 P f5ec1f95ec4544deb9e4ba19316d3e02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5ec1f95ec4544deb9e4ba19316d3e02, State: Initialized, Role: FOLLOWER
I20260620 18:35:43.615634 24919 consensus_queue.cc:260] T 0266d73c65cb497e9dc91469a90ae127 P f5ec1f95ec4544deb9e4ba19316d3e02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ec1f95ec4544deb9e4ba19316d3e02" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 39519 } } peers { permanent_uuid: "819af79e262244839ea1ff6ed5aec230" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 33193 } } peers { permanent_uuid: "a6facfe4b14c490e8cc97f3d40b6d4df" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35601 } }
I20260620 18:35:43.617654 24919 ts_tablet_manager.cc:1428] T 0266d73c65cb497e9dc91469a90ae127 P f5ec1f95ec4544deb9e4ba19316d3e02: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260620 18:35:43.621189 24760 heartbeater.cc:503] Master 127.22.37.62:38817 was elected leader, sending a full tablet report...
I20260620 18:35:43.620971 24922 raft_consensus.cc:348] T 0266d73c65cb497e9dc91469a90ae127 P a6facfe4b14c490e8cc97f3d40b6d4df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ec1f95ec4544deb9e4ba19316d3e02" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 39519 } } peers { permanent_uuid: "819af79e262244839ea1ff6ed5aec230" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 33193 } } peers { permanent_uuid: "a6facfe4b14c490e8cc97f3d40b6d4df" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35601 } }
I20260620 18:35:43.621335 24922 raft_consensus.cc:374] T 0266d73c65cb497e9dc91469a90ae127 P a6facfe4b14c490e8cc97f3d40b6d4df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:35:43.621451 24922 raft_consensus.cc:729] T 0266d73c65cb497e9dc91469a90ae127 P a6facfe4b14c490e8cc97f3d40b6d4df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6facfe4b14c490e8cc97f3d40b6d4df, State: Initialized, Role: FOLLOWER
I20260620 18:35:43.621970 24922 consensus_queue.cc:260] T 0266d73c65cb497e9dc91469a90ae127 P a6facfe4b14c490e8cc97f3d40b6d4df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ec1f95ec4544deb9e4ba19316d3e02" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 39519 } } peers { permanent_uuid: "819af79e262244839ea1ff6ed5aec230" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 33193 } } peers { permanent_uuid: "a6facfe4b14c490e8cc97f3d40b6d4df" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35601 } }
I20260620 18:35:43.622130 24918 ts_tablet_manager.cc:1428] T 0266d73c65cb497e9dc91469a90ae127 P 819af79e262244839ea1ff6ed5aec230: Time spent starting tablet: real 0.020s user 0.012s sys 0.008s
I20260620 18:35:43.622951 24899 heartbeater.cc:503] Master 127.22.37.62:38817 was elected leader, sending a full tablet report...
I20260620 18:35:43.623533 24922 ts_tablet_manager.cc:1428] T 0266d73c65cb497e9dc91469a90ae127 P a6facfe4b14c490e8cc97f3d40b6d4df: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
W20260620 18:35:43.695652 24622 tablet.cc:2369] T 0266d73c65cb497e9dc91469a90ae127 P f5ec1f95ec4544deb9e4ba19316d3e02: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:35:43.715263 24761 tablet.cc:2369] T 0266d73c65cb497e9dc91469a90ae127 P 819af79e262244839ea1ff6ed5aec230: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:35:43.738082 24900 tablet.cc:2369] T 0266d73c65cb497e9dc91469a90ae127 P a6facfe4b14c490e8cc97f3d40b6d4df: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:35:43.869485 24925 raft_consensus.cc:482] T 0266d73c65cb497e9dc91469a90ae127 P f5ec1f95ec4544deb9e4ba19316d3e02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:35:43.869884 24925 raft_consensus.cc:504] T 0266d73c65cb497e9dc91469a90ae127 P f5ec1f95ec4544deb9e4ba19316d3e02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ec1f95ec4544deb9e4ba19316d3e02" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 39519 } } peers { permanent_uuid: "819af79e262244839ea1ff6ed5aec230" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 33193 } } peers { permanent_uuid: "a6facfe4b14c490e8cc97f3d40b6d4df" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35601 } }
I20260620 18:35:43.871639 24925 leader_election.cc:288] T 0266d73c65cb497e9dc91469a90ae127 P f5ec1f95ec4544deb9e4ba19316d3e02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 819af79e262244839ea1ff6ed5aec230 (127.22.37.2:33193), a6facfe4b14c490e8cc97f3d40b6d4df (127.22.37.3:35601)
I20260620 18:35:43.881103 24915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:43.872084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.1:48493 (local address 127.22.37.2:33193)
0620 18:35:43.872296 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:43.872306 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:43.873212 (+ 906us) server_negotiation.cc:407] Connection header received
0620 18:35:43.873308 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:43.873315 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:43.873437 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:43.873527 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:35:43.874368 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:43.875512 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:43.878628 (+ 3116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:43.879879 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:43.880010 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:43.880123 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:43.880701 (+ 578us) server_negotiation.cc:299] Negotiation successful
0620 18:35:43.880833 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 18:35:43.881112 24917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:43.872117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.1:33695 (local address 127.22.37.3:35601)
0620 18:35:43.872357 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:43.872366 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:43.873363 (+ 997us) server_negotiation.cc:407] Connection header received
0620 18:35:43.873605 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:43.873612 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:43.873684 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:43.873747 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:35:43.874545 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:43.875551 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:43.878615 (+ 3064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:43.879878 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:43.880010 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:43.880127 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:43.880701 (+ 574us) server_negotiation.cc:299] Negotiation successful
0620 18:35:43.880833 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 18:35:43.881485 24928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:43.872073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.3:35601 (local address 127.22.37.1:33695)
0620 18:35:43.873246 (+ 1173us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:43.873268 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:43.873441 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:43.873956 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:43.873962 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:43.873981 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:35:43.874409 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:43.874418 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:43.875700 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:43.875705 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:43.878440 (+ 2735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:43.878456 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:43.880150 (+ 1694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:43.880155 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:43.880433 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:43.880566 (+ 133us) client_negotiation.cc:769] Sending connection context
0620 18:35:43.880666 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 18:35:43.880920 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":936,"mutex_wait_us":337,"thread_start_us":500,"threads_started":1}
I20260620 18:35:43.881508 24927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:43.871995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.2:33193 (local address 127.22.37.1:48493)
0620 18:35:43.873001 (+ 1006us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:43.873018 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:43.873151 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:43.873729 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:43.873735 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:43.873759 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:35:43.874213 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:43.874231 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:43.875699 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:43.875705 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:43.878441 (+ 2736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:43.878456 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:43.880138 (+ 1682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:43.880145 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:43.880430 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:43.880572 (+ 142us) client_negotiation.cc:769] Sending connection context
0620 18:35:43.880666 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 18:35:43.880920 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":882,"mutex_wait_us":257,"thread_start_us":505,"threads_started":1}
I20260620 18:35:43.882484 24854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0266d73c65cb497e9dc91469a90ae127" candidate_uuid: "f5ec1f95ec4544deb9e4ba19316d3e02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6facfe4b14c490e8cc97f3d40b6d4df" is_pre_election: true
I20260620 18:35:43.882489 24715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0266d73c65cb497e9dc91469a90ae127" candidate_uuid: "f5ec1f95ec4544deb9e4ba19316d3e02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819af79e262244839ea1ff6ed5aec230" is_pre_election: true
I20260620 18:35:43.882972 24715 raft_consensus.cc:2454] T 0266d73c65cb497e9dc91469a90ae127 P 819af79e262244839ea1ff6ed5aec230 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5ec1f95ec4544deb9e4ba19316d3e02 in term 0.
I20260620 18:35:43.882969 24854 raft_consensus.cc:2454] T 0266d73c65cb497e9dc91469a90ae127 P a6facfe4b14c490e8cc97f3d40b6d4df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5ec1f95ec4544deb9e4ba19316d3e02 in term 0.
I20260620 18:35:43.883631 24502 leader_election.cc:302] T 0266d73c65cb497e9dc91469a90ae127 P f5ec1f95ec4544deb9e4ba19316d3e02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6facfe4b14c490e8cc97f3d40b6d4df, f5ec1f95ec4544deb9e4ba19316d3e02; no voters:
I20260620 18:35:43.884052 24925 raft_consensus.cc:2781] T 0266d73c65cb497e9dc91469a90ae127 P f5ec1f95ec4544deb9e4ba19316d3e02 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:35:43.884181 24925 raft_consensus.cc:482] T 0266d73c65cb497e9dc91469a90ae127 P f5ec1f95ec4544deb9e4ba19316d3e02 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:35:43.884253 24925 raft_consensus.cc:3037] T 0266d73c65cb497e9dc91469a90ae127 P f5ec1f95ec4544deb9e4ba19316d3e02 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:35:43.886571 24925 raft_consensus.cc:504] T 0266d73c65cb497e9dc91469a90ae127 P f5ec1f95ec4544deb9e4ba19316d3e02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ec1f95ec4544deb9e4ba19316d3e02" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 39519 } } peers { permanent_uuid: "819af79e262244839ea1ff6ed5aec230" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 33193 } } peers { permanent_uuid: "a6facfe4b14c490e8cc97f3d40b6d4df" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35601 } }
I20260620 18:35:43.887172 24925 leader_election.cc:288] T 0266d73c65cb497e9dc91469a90ae127 P f5ec1f95ec4544deb9e4ba19316d3e02 [CANDIDATE]: Term 1 election: Requested vote from peers 819af79e262244839ea1ff6ed5aec230 (127.22.37.2:33193), a6facfe4b14c490e8cc97f3d40b6d4df (127.22.37.3:35601)
I20260620 18:35:43.887516 24715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0266d73c65cb497e9dc91469a90ae127" candidate_uuid: "f5ec1f95ec4544deb9e4ba19316d3e02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819af79e262244839ea1ff6ed5aec230"
I20260620 18:35:43.887633 24854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0266d73c65cb497e9dc91469a90ae127" candidate_uuid: "f5ec1f95ec4544deb9e4ba19316d3e02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6facfe4b14c490e8cc97f3d40b6d4df"
I20260620 18:35:43.887701 24715 raft_consensus.cc:3037] T 0266d73c65cb497e9dc91469a90ae127 P 819af79e262244839ea1ff6ed5aec230 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:35:43.887847 24854 raft_consensus.cc:3037] T 0266d73c65cb497e9dc91469a90ae127 P a6facfe4b14c490e8cc97f3d40b6d4df [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:35:43.889861 24715 raft_consensus.cc:2454] T 0266d73c65cb497e9dc91469a90ae127 P 819af79e262244839ea1ff6ed5aec230 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5ec1f95ec4544deb9e4ba19316d3e02 in term 1.
I20260620 18:35:43.890019 24854 raft_consensus.cc:2454] T 0266d73c65cb497e9dc91469a90ae127 P a6facfe4b14c490e8cc97f3d40b6d4df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5ec1f95ec4544deb9e4ba19316d3e02 in term 1.
I20260620 18:35:43.890269 24501 leader_election.cc:302] T 0266d73c65cb497e9dc91469a90ae127 P f5ec1f95ec4544deb9e4ba19316d3e02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 819af79e262244839ea1ff6ed5aec230, f5ec1f95ec4544deb9e4ba19316d3e02; no voters:
I20260620 18:35:43.890552 24925 raft_consensus.cc:2781] T 0266d73c65cb497e9dc91469a90ae127 P f5ec1f95ec4544deb9e4ba19316d3e02 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:35:43.890952 24925 raft_consensus.cc:686] T 0266d73c65cb497e9dc91469a90ae127 P f5ec1f95ec4544deb9e4ba19316d3e02 [term 1 LEADER]: Becoming Leader. State: Replica: f5ec1f95ec4544deb9e4ba19316d3e02, State: Running, Role: LEADER
I20260620 18:35:43.891396 24925 consensus_queue.cc:237] T 0266d73c65cb497e9dc91469a90ae127 P f5ec1f95ec4544deb9e4ba19316d3e02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ec1f95ec4544deb9e4ba19316d3e02" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 39519 } } peers { permanent_uuid: "819af79e262244839ea1ff6ed5aec230" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 33193 } } peers { permanent_uuid: "a6facfe4b14c490e8cc97f3d40b6d4df" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35601 } }
I20260620 18:35:43.895601 24413 catalog_manager.cc:5495] T 0266d73c65cb497e9dc91469a90ae127 P f5ec1f95ec4544deb9e4ba19316d3e02 reported cstate change: term changed from 0 to 1, leader changed from <none> to f5ec1f95ec4544deb9e4ba19316d3e02 (127.22.37.1). New cstate: current_term: 1 leader_uuid: "f5ec1f95ec4544deb9e4ba19316d3e02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ec1f95ec4544deb9e4ba19316d3e02" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 39519 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "819af79e262244839ea1ff6ed5aec230" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 33193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6facfe4b14c490e8cc97f3d40b6d4df" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35601 } health_report { overall_health: UNKNOWN } } }
Jun 20 18:35:43 dist-test-slave-5lgp krb5kdc[24369](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980541, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.62@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM
I20260620 18:35:43.965996 24936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:43.952066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:38817 (local address 127.0.0.1:39062)
0620 18:35:43.952709 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:43.952725 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:43.952897 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:43.953182 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:43.953185 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:43.953977 (+ 792us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:43.954288 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:43.954297 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:43.955392 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:43.955395 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:43.956366 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:43.956376 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:43.957225 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:43.957229 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:43.957355 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:43.958018 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:35:43.958035 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:43.960165 (+ 2130us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:35:43.963607 (+ 3442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:43.963617 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:43.963622 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:43.964015 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:43.964409 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:43.964412 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:43.964415 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:43.964537 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:43.965088 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:35:43.965094 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:35:43.965241 (+ 147us) client_negotiation.cc:769] Sending connection context
0620 18:35:43.965425 (+ 184us) client_negotiation.cc:240] Negotiation successful
0620 18:35:43.965650 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":387,"threads_started":1}
I20260620 18:35:43.966442 24509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:43.951997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39062 (local address 127.22.37.62:38817)
0620 18:35:43.952220 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:43.952229 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:43.952840 (+ 611us) server_negotiation.cc:407] Connection header received
0620 18:35:43.952999 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:43.953002 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:43.953061 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:43.953147 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:43.954478 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:43.955271 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:43.956523 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:43.957056 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:43.957172 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:43.960332 (+ 3160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:43.960337 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:43.960340 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:43.960368 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:43.963465 (+ 3097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:43.964134 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:43.964139 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:43.964142 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:43.964299 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:43.964632 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:43.964637 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:43.964640 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:43.964963 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:43.965128 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:43.965516 (+ 388us) server_negotiation.cc:299] Negotiation successful
0620 18:35:43.965799 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":61}
I20260620 18:35:43.971714 24413 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.22.37.62@KRBTEST.COM'} at 127.0.0.1:39062:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 18:35:43.972700 24413 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:35:43.979452 24551 tablet_service.cc:1467] Processing CreateTablet for tablet 8d1243cb18af48b0b4d1c78123dfaaa8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=402518e07f6a4193af8981a73e547ec9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 18:35:43.979694 24833 tablet_service.cc:1467] Processing CreateTablet for tablet 8d1243cb18af48b0b4d1c78123dfaaa8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=402518e07f6a4193af8981a73e547ec9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 18:35:43.979691 24690 tablet_service.cc:1467] Processing CreateTablet for tablet 8d1243cb18af48b0b4d1c78123dfaaa8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=402518e07f6a4193af8981a73e547ec9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 18:35:43.980491 24551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d1243cb18af48b0b4d1c78123dfaaa8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:35:43.980530 24690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d1243cb18af48b0b4d1c78123dfaaa8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:35:43.980535 24833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d1243cb18af48b0b4d1c78123dfaaa8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:35:43.986205 24919 tablet_bootstrap.cc:492] T 8d1243cb18af48b0b4d1c78123dfaaa8 P f5ec1f95ec4544deb9e4ba19316d3e02: Bootstrap starting.
I20260620 18:35:43.987598 24922 tablet_bootstrap.cc:492] T 8d1243cb18af48b0b4d1c78123dfaaa8 P a6facfe4b14c490e8cc97f3d40b6d4df: Bootstrap starting.
I20260620 18:35:43.987632 24919 tablet_bootstrap.cc:654] T 8d1243cb18af48b0b4d1c78123dfaaa8 P f5ec1f95ec4544deb9e4ba19316d3e02: Neither blocks nor log segments found. Creating new log.
I20260620 18:35:43.988107 24918 tablet_bootstrap.cc:492] T 8d1243cb18af48b0b4d1c78123dfaaa8 P 819af79e262244839ea1ff6ed5aec230: Bootstrap starting.
I20260620 18:35:43.989530 24918 tablet_bootstrap.cc:654] T 8d1243cb18af48b0b4d1c78123dfaaa8 P 819af79e262244839ea1ff6ed5aec230: Neither blocks nor log segments found. Creating new log.
I20260620 18:35:43.989564 24922 tablet_bootstrap.cc:654] T 8d1243cb18af48b0b4d1c78123dfaaa8 P a6facfe4b14c490e8cc97f3d40b6d4df: Neither blocks nor log segments found. Creating new log.
I20260620 18:35:43.998948 24918 tablet_bootstrap.cc:492] T 8d1243cb18af48b0b4d1c78123dfaaa8 P 819af79e262244839ea1ff6ed5aec230: No bootstrap required, opened a new log
I20260620 18:35:43.999132 24918 ts_tablet_manager.cc:1397] T 8d1243cb18af48b0b4d1c78123dfaaa8 P 819af79e262244839ea1ff6ed5aec230: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.006s
I20260620 18:35:43.999131 24919 tablet_bootstrap.cc:492] T 8d1243cb18af48b0b4d1c78123dfaaa8 P f5ec1f95ec4544deb9e4ba19316d3e02: No bootstrap required, opened a new log
I20260620 18:35:43.999150 24922 tablet_bootstrap.cc:492] T 8d1243cb18af48b0b4d1c78123dfaaa8 P a6facfe4b14c490e8cc97f3d40b6d4df: No bootstrap required, opened a new log
I20260620 18:35:43.999296 24922 ts_tablet_manager.cc:1397] T 8d1243cb18af48b0b4d1c78123dfaaa8 P a6facfe4b14c490e8cc97f3d40b6d4df: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.010s
I20260620 18:35:43.999315 24919 ts_tablet_manager.cc:1397] T 8d1243cb18af48b0b4d1c78123dfaaa8 P f5ec1f95ec4544deb9e4ba19316d3e02: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.008s
I20260620 18:35:43.999852 24918 raft_consensus.cc:348] T 8d1243cb18af48b0b4d1c78123dfaaa8 P 819af79e262244839ea1ff6ed5aec230 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ec1f95ec4544deb9e4ba19316d3e02" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 39519 } } peers { permanent_uuid: "819af79e262244839ea1ff6ed5aec230" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 33193 } } peers { permanent_uuid: "a6facfe4b14c490e8cc97f3d40b6d4df" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35601 } }
I20260620 18:35:44.000072 24918 raft_consensus.cc:374] T 8d1243cb18af48b0b4d1c78123dfaaa8 P 819af79e262244839ea1ff6ed5aec230 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:35:44.000165 24918 raft_consensus.cc:729] T 8d1243cb18af48b0b4d1c78123dfaaa8 P 819af79e262244839ea1ff6ed5aec230 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819af79e262244839ea1ff6ed5aec230, State: Initialized, Role: FOLLOWER
I20260620 18:35:44.000062 24922 raft_consensus.cc:348] T 8d1243cb18af48b0b4d1c78123dfaaa8 P a6facfe4b14c490e8cc97f3d40b6d4df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ec1f95ec4544deb9e4ba19316d3e02" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 39519 } } peers { permanent_uuid: "819af79e262244839ea1ff6ed5aec230" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 33193 } } peers { permanent_uuid: "a6facfe4b14c490e8cc97f3d40b6d4df" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35601 } }
I20260620 18:35:44.000077 24919 raft_consensus.cc:348] T 8d1243cb18af48b0b4d1c78123dfaaa8 P f5ec1f95ec4544deb9e4ba19316d3e02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ec1f95ec4544deb9e4ba19316d3e02" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 39519 } } peers { permanent_uuid: "819af79e262244839ea1ff6ed5aec230" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 33193 } } peers { permanent_uuid: "a6facfe4b14c490e8cc97f3d40b6d4df" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35601 } }
I20260620 18:35:44.000296 24919 raft_consensus.cc:374] T 8d1243cb18af48b0b4d1c78123dfaaa8 P f5ec1f95ec4544deb9e4ba19316d3e02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:35:44.000296 24922 raft_consensus.cc:374] T 8d1243cb18af48b0b4d1c78123dfaaa8 P a6facfe4b14c490e8cc97f3d40b6d4df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:35:44.000397 24922 raft_consensus.cc:729] T 8d1243cb18af48b0b4d1c78123dfaaa8 P a6facfe4b14c490e8cc97f3d40b6d4df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6facfe4b14c490e8cc97f3d40b6d4df, State: Initialized, Role: FOLLOWER
I20260620 18:35:44.000396 24919 raft_consensus.cc:729] T 8d1243cb18af48b0b4d1c78123dfaaa8 P f5ec1f95ec4544deb9e4ba19316d3e02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5ec1f95ec4544deb9e4ba19316d3e02, State: Initialized, Role: FOLLOWER
I20260620 18:35:44.000437 24918 consensus_queue.cc:260] T 8d1243cb18af48b0b4d1c78123dfaaa8 P 819af79e262244839ea1ff6ed5aec230 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ec1f95ec4544deb9e4ba19316d3e02" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 39519 } } peers { permanent_uuid: "819af79e262244839ea1ff6ed5aec230" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 33193 } } peers { permanent_uuid: "a6facfe4b14c490e8cc97f3d40b6d4df" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35601 } }
I20260620 18:35:44.000648 24922 consensus_queue.cc:260] T 8d1243cb18af48b0b4d1c78123dfaaa8 P a6facfe4b14c490e8cc97f3d40b6d4df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ec1f95ec4544deb9e4ba19316d3e02" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 39519 } } peers { permanent_uuid: "819af79e262244839ea1ff6ed5aec230" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 33193 } } peers { permanent_uuid: "a6facfe4b14c490e8cc97f3d40b6d4df" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35601 } }
I20260620 18:35:44.000902 24919 consensus_queue.cc:260] T 8d1243cb18af48b0b4d1c78123dfaaa8 P f5ec1f95ec4544deb9e4ba19316d3e02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ec1f95ec4544deb9e4ba19316d3e02" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 39519 } } peers { permanent_uuid: "819af79e262244839ea1ff6ed5aec230" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 33193 } } peers { permanent_uuid: "a6facfe4b14c490e8cc97f3d40b6d4df" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35601 } }
I20260620 18:35:44.001134 24918 ts_tablet_manager.cc:1428] T 8d1243cb18af48b0b4d1c78123dfaaa8 P 819af79e262244839ea1ff6ed5aec230: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260620 18:35:44.001184 24922 ts_tablet_manager.cc:1428] T 8d1243cb18af48b0b4d1c78123dfaaa8 P a6facfe4b14c490e8cc97f3d40b6d4df: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260620 18:35:44.000973 24924 tablet_replica.cc:439] T 8d1243cb18af48b0b4d1c78123dfaaa8 P 819af79e262244839ea1ff6ed5aec230: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ec1f95ec4544deb9e4ba19316d3e02" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 39519 } } peers { permanent_uuid: "819af79e262244839ea1ff6ed5aec230" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 33193 } } peers { permanent_uuid: "a6facfe4b14c490e8cc97f3d40b6d4df" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35601 } } }
I20260620 18:35:44.001413 24924 tablet_replica.cc:442] T 8d1243cb18af48b0b4d1c78123dfaaa8 P 819af79e262244839ea1ff6ed5aec230: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:35:44.001506 24919 ts_tablet_manager.cc:1428] T 8d1243cb18af48b0b4d1c78123dfaaa8 P f5ec1f95ec4544deb9e4ba19316d3e02: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260620 18:35:44.002096 24925 tablet_replica.cc:439] T 8d1243cb18af48b0b4d1c78123dfaaa8 P f5ec1f95ec4544deb9e4ba19316d3e02: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ec1f95ec4544deb9e4ba19316d3e02" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 39519 } } peers { permanent_uuid: "819af79e262244839ea1ff6ed5aec230" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 33193 } } peers { permanent_uuid: "a6facfe4b14c490e8cc97f3d40b6d4df" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35601 } } }
I20260620 18:35:44.002353 24925 tablet_replica.cc:442] T 8d1243cb18af48b0b4d1c78123dfaaa8 P f5ec1f95ec4544deb9e4ba19316d3e02: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:35:44.002543 24926 tablet_replica.cc:439] T 8d1243cb18af48b0b4d1c78123dfaaa8 P a6facfe4b14c490e8cc97f3d40b6d4df: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ec1f95ec4544deb9e4ba19316d3e02" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 39519 } } peers { permanent_uuid: "819af79e262244839ea1ff6ed5aec230" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 33193 } } peers { permanent_uuid: "a6facfe4b14c490e8cc97f3d40b6d4df" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35601 } } }
I20260620 18:35:44.002812 24926 tablet_replica.cc:442] T 8d1243cb18af48b0b4d1c78123dfaaa8 P a6facfe4b14c490e8cc97f3d40b6d4df: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:35:44.019873 24925 raft_consensus.cc:482] T 8d1243cb18af48b0b4d1c78123dfaaa8 P f5ec1f95ec4544deb9e4ba19316d3e02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:35:44.020069 24925 raft_consensus.cc:504] T 8d1243cb18af48b0b4d1c78123dfaaa8 P f5ec1f95ec4544deb9e4ba19316d3e02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ec1f95ec4544deb9e4ba19316d3e02" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 39519 } } peers { permanent_uuid: "819af79e262244839ea1ff6ed5aec230" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 33193 } } peers { permanent_uuid: "a6facfe4b14c490e8cc97f3d40b6d4df" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35601 } }
I20260620 18:35:44.020661 24925 leader_election.cc:288] T 8d1243cb18af48b0b4d1c78123dfaaa8 P f5ec1f95ec4544deb9e4ba19316d3e02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 819af79e262244839ea1ff6ed5aec230 (127.22.37.2:33193), a6facfe4b14c490e8cc97f3d40b6d4df (127.22.37.3:35601)
I20260620 18:35:44.021116 24854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d1243cb18af48b0b4d1c78123dfaaa8" candidate_uuid: "f5ec1f95ec4544deb9e4ba19316d3e02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6facfe4b14c490e8cc97f3d40b6d4df" is_pre_election: true
I20260620 18:35:44.021118 24715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d1243cb18af48b0b4d1c78123dfaaa8" candidate_uuid: "f5ec1f95ec4544deb9e4ba19316d3e02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819af79e262244839ea1ff6ed5aec230" is_pre_election: true
I20260620 18:35:44.021355 24854 raft_consensus.cc:2454] T 8d1243cb18af48b0b4d1c78123dfaaa8 P a6facfe4b14c490e8cc97f3d40b6d4df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5ec1f95ec4544deb9e4ba19316d3e02 in term 0.
I20260620 18:35:44.021400 24715 raft_consensus.cc:2454] T 8d1243cb18af48b0b4d1c78123dfaaa8 P 819af79e262244839ea1ff6ed5aec230 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5ec1f95ec4544deb9e4ba19316d3e02 in term 0.
I20260620 18:35:44.021782 24502 leader_election.cc:302] T 8d1243cb18af48b0b4d1c78123dfaaa8 P f5ec1f95ec4544deb9e4ba19316d3e02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6facfe4b14c490e8cc97f3d40b6d4df, f5ec1f95ec4544deb9e4ba19316d3e02; no voters:
I20260620 18:35:44.022043 24925 raft_consensus.cc:2781] T 8d1243cb18af48b0b4d1c78123dfaaa8 P f5ec1f95ec4544deb9e4ba19316d3e02 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:35:44.022173 24925 raft_consensus.cc:482] T 8d1243cb18af48b0b4d1c78123dfaaa8 P f5ec1f95ec4544deb9e4ba19316d3e02 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:35:44.022302 24925 raft_consensus.cc:3037] T 8d1243cb18af48b0b4d1c78123dfaaa8 P f5ec1f95ec4544deb9e4ba19316d3e02 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:35:44.024370 24925 raft_consensus.cc:504] T 8d1243cb18af48b0b4d1c78123dfaaa8 P f5ec1f95ec4544deb9e4ba19316d3e02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ec1f95ec4544deb9e4ba19316d3e02" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 39519 } } peers { permanent_uuid: "819af79e262244839ea1ff6ed5aec230" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 33193 } } peers { permanent_uuid: "a6facfe4b14c490e8cc97f3d40b6d4df" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35601 } }
I20260620 18:35:44.024885 24925 leader_election.cc:288] T 8d1243cb18af48b0b4d1c78123dfaaa8 P f5ec1f95ec4544deb9e4ba19316d3e02 [CANDIDATE]: Term 1 election: Requested vote from peers 819af79e262244839ea1ff6ed5aec230 (127.22.37.2:33193), a6facfe4b14c490e8cc97f3d40b6d4df (127.22.37.3:35601)
I20260620 18:35:44.025285 24715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d1243cb18af48b0b4d1c78123dfaaa8" candidate_uuid: "f5ec1f95ec4544deb9e4ba19316d3e02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819af79e262244839ea1ff6ed5aec230"
I20260620 18:35:44.025286 24854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d1243cb18af48b0b4d1c78123dfaaa8" candidate_uuid: "f5ec1f95ec4544deb9e4ba19316d3e02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6facfe4b14c490e8cc97f3d40b6d4df"
I20260620 18:35:44.025519 24854 raft_consensus.cc:3037] T 8d1243cb18af48b0b4d1c78123dfaaa8 P a6facfe4b14c490e8cc97f3d40b6d4df [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:35:44.025521 24715 raft_consensus.cc:3037] T 8d1243cb18af48b0b4d1c78123dfaaa8 P 819af79e262244839ea1ff6ed5aec230 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:35:44.028407 24854 raft_consensus.cc:2454] T 8d1243cb18af48b0b4d1c78123dfaaa8 P a6facfe4b14c490e8cc97f3d40b6d4df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5ec1f95ec4544deb9e4ba19316d3e02 in term 1.
I20260620 18:35:44.028407 24715 raft_consensus.cc:2454] T 8d1243cb18af48b0b4d1c78123dfaaa8 P 819af79e262244839ea1ff6ed5aec230 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5ec1f95ec4544deb9e4ba19316d3e02 in term 1.
I20260620 18:35:44.028801 24502 leader_election.cc:302] T 8d1243cb18af48b0b4d1c78123dfaaa8 P f5ec1f95ec4544deb9e4ba19316d3e02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6facfe4b14c490e8cc97f3d40b6d4df, f5ec1f95ec4544deb9e4ba19316d3e02; no voters:
I20260620 18:35:44.029028 24925 raft_consensus.cc:2781] T 8d1243cb18af48b0b4d1c78123dfaaa8 P f5ec1f95ec4544deb9e4ba19316d3e02 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:35:44.029241 24925 raft_consensus.cc:686] T 8d1243cb18af48b0b4d1c78123dfaaa8 P f5ec1f95ec4544deb9e4ba19316d3e02 [term 1 LEADER]: Becoming Leader. State: Replica: f5ec1f95ec4544deb9e4ba19316d3e02, State: Running, Role: LEADER
I20260620 18:35:44.029531 24925 consensus_queue.cc:237] T 8d1243cb18af48b0b4d1c78123dfaaa8 P f5ec1f95ec4544deb9e4ba19316d3e02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ec1f95ec4544deb9e4ba19316d3e02" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 39519 } } peers { permanent_uuid: "819af79e262244839ea1ff6ed5aec230" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 33193 } } peers { permanent_uuid: "a6facfe4b14c490e8cc97f3d40b6d4df" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35601 } }
I20260620 18:35:44.031133 24929 tablet_replica.cc:439] T 8d1243cb18af48b0b4d1c78123dfaaa8 P f5ec1f95ec4544deb9e4ba19316d3e02: TxnStatusTablet state changed. Reason: New leader f5ec1f95ec4544deb9e4ba19316d3e02. Latest consensus state: current_term: 1 leader_uuid: "f5ec1f95ec4544deb9e4ba19316d3e02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ec1f95ec4544deb9e4ba19316d3e02" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 39519 } } peers { permanent_uuid: "819af79e262244839ea1ff6ed5aec230" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 33193 } } peers { permanent_uuid: "a6facfe4b14c490e8cc97f3d40b6d4df" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35601 } } }
I20260620 18:35:44.031450 24929 tablet_replica.cc:442] T 8d1243cb18af48b0b4d1c78123dfaaa8 P f5ec1f95ec4544deb9e4ba19316d3e02: This TxnStatusTablet replica's current role is: LEADER
I20260620 18:35:44.032060 24941 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 18:35:44.032404 24941 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 18:35:44.033048 24413 catalog_manager.cc:5495] T 8d1243cb18af48b0b4d1c78123dfaaa8 P f5ec1f95ec4544deb9e4ba19316d3e02 reported cstate change: term changed from 0 to 1, leader changed from <none> to f5ec1f95ec4544deb9e4ba19316d3e02 (127.22.37.1). New cstate: current_term: 1 leader_uuid: "f5ec1f95ec4544deb9e4ba19316d3e02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ec1f95ec4544deb9e4ba19316d3e02" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 39519 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "819af79e262244839ea1ff6ed5aec230" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 33193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6facfe4b14c490e8cc97f3d40b6d4df" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35601 } health_report { overall_health: UNKNOWN } } }
I20260620 18:35:44.056221 24936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:44.050586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.1:39519 (local address 127.0.0.1:54846)
0620 18:35:44.050843 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:44.050862 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:44.051004 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:44.051497 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:44.051502 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:44.051515 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 18:35:44.051694 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:44.051703 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:44.052775 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:44.052780 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:44.053883 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:44.053894 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:44.054583 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:44.054589 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:44.054765 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:44.054817 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 18:35:44.055662 (+ 845us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 18:35:44.055682 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 18:35:44.055783 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 18:35:44.055936 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260620 18:35:44.056195 24913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:44.050707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54846 (local address 127.22.37.1:39519)
0620 18:35:44.050894 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:44.050903 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:44.050923 (+ 20us) server_negotiation.cc:407] Connection header received
0620 18:35:44.051220 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:44.051224 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:44.051329 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:44.051479 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:35:44.051821 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:44.052618 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:44.054015 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:44.054439 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:44.054568 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:44.055018 (+ 450us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:35:44.055543 (+ 525us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:35:44.055693 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:44.055807 (+ 114us) server_negotiation.cc:299] Negotiation successful
0620 18:35:44.055887 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 18:35:44.065302 24715 raft_consensus.cc:1261] T 8d1243cb18af48b0b4d1c78123dfaaa8 P 819af79e262244839ea1ff6ed5aec230 [term 1 FOLLOWER]: Refusing update from remote peer f5ec1f95ec4544deb9e4ba19316d3e02: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:35:44.065302 24854 raft_consensus.cc:1261] T 8d1243cb18af48b0b4d1c78123dfaaa8 P a6facfe4b14c490e8cc97f3d40b6d4df [term 1 FOLLOWER]: Refusing update from remote peer f5ec1f95ec4544deb9e4ba19316d3e02: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:35:44.066289 24929 consensus_queue.cc:1035] T 8d1243cb18af48b0b4d1c78123dfaaa8 P f5ec1f95ec4544deb9e4ba19316d3e02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "819af79e262244839ea1ff6ed5aec230" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:35:44.066633 24925 consensus_queue.cc:1035] T 8d1243cb18af48b0b4d1c78123dfaaa8 P f5ec1f95ec4544deb9e4ba19316d3e02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6facfe4b14c490e8cc97f3d40b6d4df" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:35:44.079378 24926 tablet_replica.cc:439] T 8d1243cb18af48b0b4d1c78123dfaaa8 P a6facfe4b14c490e8cc97f3d40b6d4df: TxnStatusTablet state changed. Reason: New leader f5ec1f95ec4544deb9e4ba19316d3e02. Latest consensus state: current_term: 1 leader_uuid: "f5ec1f95ec4544deb9e4ba19316d3e02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ec1f95ec4544deb9e4ba19316d3e02" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 39519 } } peers { permanent_uuid: "819af79e262244839ea1ff6ed5aec230" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 33193 } } peers { permanent_uuid: "a6facfe4b14c490e8cc97f3d40b6d4df" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35601 } } }
I20260620 18:35:44.079670 24926 tablet_replica.cc:442] T 8d1243cb18af48b0b4d1c78123dfaaa8 P a6facfe4b14c490e8cc97f3d40b6d4df: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:35:44.082646 24925 tablet_replica.cc:439] T 8d1243cb18af48b0b4d1c78123dfaaa8 P f5ec1f95ec4544deb9e4ba19316d3e02: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f5ec1f95ec4544deb9e4ba19316d3e02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ec1f95ec4544deb9e4ba19316d3e02" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 39519 } } peers { permanent_uuid: "819af79e262244839ea1ff6ed5aec230" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 33193 } } peers { permanent_uuid: "a6facfe4b14c490e8cc97f3d40b6d4df" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35601 } } }
I20260620 18:35:44.082947 24925 tablet_replica.cc:442] T 8d1243cb18af48b0b4d1c78123dfaaa8 P f5ec1f95ec4544deb9e4ba19316d3e02: This TxnStatusTablet replica's current role is: LEADER
I20260620 18:35:44.091499 24924 tablet_replica.cc:439] T 8d1243cb18af48b0b4d1c78123dfaaa8 P 819af79e262244839ea1ff6ed5aec230: TxnStatusTablet state changed. Reason: New leader f5ec1f95ec4544deb9e4ba19316d3e02. Latest consensus state: current_term: 1 leader_uuid: "f5ec1f95ec4544deb9e4ba19316d3e02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ec1f95ec4544deb9e4ba19316d3e02" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 39519 } } peers { permanent_uuid: "819af79e262244839ea1ff6ed5aec230" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 33193 } } peers { permanent_uuid: "a6facfe4b14c490e8cc97f3d40b6d4df" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35601 } } }
I20260620 18:35:44.091779 24924 tablet_replica.cc:442] T 8d1243cb18af48b0b4d1c78123dfaaa8 P 819af79e262244839ea1ff6ed5aec230: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:35:44.094924 24929 tablet_replica.cc:439] T 8d1243cb18af48b0b4d1c78123dfaaa8 P f5ec1f95ec4544deb9e4ba19316d3e02: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f5ec1f95ec4544deb9e4ba19316d3e02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ec1f95ec4544deb9e4ba19316d3e02" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 39519 } } peers { permanent_uuid: "819af79e262244839ea1ff6ed5aec230" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 33193 } } peers { permanent_uuid: "a6facfe4b14c490e8cc97f3d40b6d4df" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35601 } } }
I20260620 18:35:44.095188 24929 tablet_replica.cc:442] T 8d1243cb18af48b0b4d1c78123dfaaa8 P f5ec1f95ec4544deb9e4ba19316d3e02: This TxnStatusTablet replica's current role is: LEADER
I20260620 18:35:44.099664 24946 mvcc.cc:205] Tried to move back new op lower bound from 7298992308480684032 to 7298992308346863616. Current Snapshot: MvccSnapshot[applied={T|T < 7298992308480684032}]
I20260620 18:35:44.101929 24924 tablet_replica.cc:439] T 8d1243cb18af48b0b4d1c78123dfaaa8 P 819af79e262244839ea1ff6ed5aec230: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f5ec1f95ec4544deb9e4ba19316d3e02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ec1f95ec4544deb9e4ba19316d3e02" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 39519 } } peers { permanent_uuid: "819af79e262244839ea1ff6ed5aec230" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 33193 } } peers { permanent_uuid: "a6facfe4b14c490e8cc97f3d40b6d4df" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35601 } } }
I20260620 18:35:44.102238 24924 tablet_replica.cc:442] T 8d1243cb18af48b0b4d1c78123dfaaa8 P 819af79e262244839ea1ff6ed5aec230: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:35:44.104101 24926 tablet_replica.cc:439] T 8d1243cb18af48b0b4d1c78123dfaaa8 P a6facfe4b14c490e8cc97f3d40b6d4df: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f5ec1f95ec4544deb9e4ba19316d3e02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ec1f95ec4544deb9e4ba19316d3e02" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 39519 } } peers { permanent_uuid: "819af79e262244839ea1ff6ed5aec230" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 33193 } } peers { permanent_uuid: "a6facfe4b14c490e8cc97f3d40b6d4df" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35601 } } }
I20260620 18:35:44.104329 24926 tablet_replica.cc:442] T 8d1243cb18af48b0b4d1c78123dfaaa8 P a6facfe4b14c490e8cc97f3d40b6d4df: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:35:44.112300 24913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:44.106358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54860 (local address 127.22.37.1:39519)
0620 18:35:44.106522 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:44.106528 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:44.106924 (+ 396us) server_negotiation.cc:407] Connection header received
0620 18:35:44.107093 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:44.107099 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:44.107190 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:44.107285 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:35:44.107685 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:44.108907 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:44.110313 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:44.110939 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:44.111066 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:44.111546 (+ 480us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:35:44.111688 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:35:44.111780 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:44.111897 (+ 117us) server_negotiation.cc:299] Negotiation successful
0620 18:35:44.111978 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 18:35:44.128695 24913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:44.122521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.1:50607 (local address 127.22.37.1:39519)
0620 18:35:44.122705 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:44.122711 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:44.123307 (+ 596us) server_negotiation.cc:407] Connection header received
0620 18:35:44.123442 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:44.123447 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:44.123521 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:44.123602 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:35:44.124042 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:44.125178 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:44.126764 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:44.127453 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:44.127586 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:44.127981 (+ 395us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:35:44.128094 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:35:44.128202 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:44.128303 (+ 101us) server_negotiation.cc:299] Negotiation successful
0620 18:35:44.128376 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 18:35:44.128697 24956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:44.122560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.1:39519 (local address 127.22.37.1:50607)
0620 18:35:44.123217 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:44.123235 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:44.123342 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:44.123646 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:44.123648 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:44.123657 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 18:35:44.123919 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:44.123926 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:44.125327 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:44.125330 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:44.126640 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:44.126651 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:44.127629 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:44.127633 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:44.127766 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:44.127812 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 18:35:44.128196 (+ 384us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 18:35:44.128204 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 18:35:44.128265 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 18:35:44.128351 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":342,"threads_started":1}
I20260620 18:35:44.137790 24854 raft_consensus.cc:1261] T 0266d73c65cb497e9dc91469a90ae127 P a6facfe4b14c490e8cc97f3d40b6d4df [term 1 FOLLOWER]: Refusing update from remote peer f5ec1f95ec4544deb9e4ba19316d3e02: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:35:44.137830 24714 raft_consensus.cc:1261] T 0266d73c65cb497e9dc91469a90ae127 P 819af79e262244839ea1ff6ed5aec230 [term 1 FOLLOWER]: Refusing update from remote peer f5ec1f95ec4544deb9e4ba19316d3e02: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:35:44.138293 24929 consensus_queue.cc:1035] T 0266d73c65cb497e9dc91469a90ae127 P f5ec1f95ec4544deb9e4ba19316d3e02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6facfe4b14c490e8cc97f3d40b6d4df" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:35:44.138546 24925 consensus_queue.cc:1035] T 0266d73c65cb497e9dc91469a90ae127 P f5ec1f95ec4544deb9e4ba19316d3e02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "819af79e262244839ea1ff6ed5aec230" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:35:44.144728 24944 mvcc.cc:205] Tried to move back new op lower bound from 7298992308783374336 to 7298992307782340608. Current Snapshot: MvccSnapshot[applied={T|T < 7298992308783374336}]
I20260620 18:35:44.194499 24413 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39058:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:35:44.194782 24413 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39058:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:35:44.198205 24413 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 518479382dc0411d87ae9ad184d6250a: Sending DeleteTablet for 3 replicas of tablet 0266d73c65cb497e9dc91469a90ae127
I20260620 18:35:44.199278 24690 tablet_service.cc:1514] Processing DeleteTablet for tablet 0266d73c65cb497e9dc91469a90ae127 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:35:44 UTC) from {username='kudu', principal='kudu/127.22.37.62@KRBTEST.COM'} at 127.0.0.1:57084
I20260620 18:35:44.199409 24550 tablet_service.cc:1514] Processing DeleteTablet for tablet 0266d73c65cb497e9dc91469a90ae127 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:35:44 UTC) from {username='kudu', principal='kudu/127.22.37.62@KRBTEST.COM'} at 127.0.0.1:54838
I20260620 18:35:44.200095 24833 tablet_service.cc:1514] Processing DeleteTablet for tablet 0266d73c65cb497e9dc91469a90ae127 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:35:44 UTC) from {username='kudu', principal='kudu/127.22.37.62@KRBTEST.COM'} at 127.0.0.1:60390
I20260620 18:35:44.200232 24961 tablet_replica.cc:330] T 0266d73c65cb497e9dc91469a90ae127 P 819af79e262244839ea1ff6ed5aec230: stopping tablet replica
I20260620 18:35:44.200822 24961 raft_consensus.cc:2229] T 0266d73c65cb497e9dc91469a90ae127 P 819af79e262244839ea1ff6ed5aec230 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:35:44.200955 24963 tablet_replica.cc:330] T 0266d73c65cb497e9dc91469a90ae127 P a6facfe4b14c490e8cc97f3d40b6d4df: stopping tablet replica
I20260620 18:35:44.201056 24413 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39058:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 18:35:44.201351 24963 raft_consensus.cc:2229] T 0266d73c65cb497e9dc91469a90ae127 P a6facfe4b14c490e8cc97f3d40b6d4df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:35:44.201331 24961 raft_consensus.cc:2258] T 0266d73c65cb497e9dc91469a90ae127 P 819af79e262244839ea1ff6ed5aec230 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:35:44.201723 24963 raft_consensus.cc:2258] T 0266d73c65cb497e9dc91469a90ae127 P a6facfe4b14c490e8cc97f3d40b6d4df [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 18:35:44.201733 24413 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:35:44.202390 24962 tablet_replica.cc:330] T 0266d73c65cb497e9dc91469a90ae127 P f5ec1f95ec4544deb9e4ba19316d3e02: stopping tablet replica
I20260620 18:35:44.202800 24962 raft_consensus.cc:2229] T 0266d73c65cb497e9dc91469a90ae127 P f5ec1f95ec4544deb9e4ba19316d3e02 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:35:44.203441 24962 raft_consensus.cc:2258] T 0266d73c65cb497e9dc91469a90ae127 P f5ec1f95ec4544deb9e4ba19316d3e02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:35:44.204746 24961 ts_tablet_manager.cc:1905] T 0266d73c65cb497e9dc91469a90ae127 P 819af79e262244839ea1ff6ed5aec230: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:35:44.204911 24963 ts_tablet_manager.cc:1905] T 0266d73c65cb497e9dc91469a90ae127 P a6facfe4b14c490e8cc97f3d40b6d4df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:35:44.209860 24550 tablet_service.cc:1467] Processing CreateTablet for tablet 4d9fb0c1a6404505bf633b7c521e662d (DEFAULT_TABLE table=test-table [id=2484c1324ccd4b64b6b8c90c2a4ad002]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:35:44.210024 24690 tablet_service.cc:1467] Processing CreateTablet for tablet 4d9fb0c1a6404505bf633b7c521e662d (DEFAULT_TABLE table=test-table [id=2484c1324ccd4b64b6b8c90c2a4ad002]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:35:44.210397 24550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d9fb0c1a6404505bf633b7c521e662d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:35:44.210417 24690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d9fb0c1a6404505bf633b7c521e662d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:35:44.213043 24961 ts_tablet_manager.cc:1918] T 0266d73c65cb497e9dc91469a90ae127 P 819af79e262244839ea1ff6ed5aec230: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:35:44.213197 24961 log.cc:1196] T 0266d73c65cb497e9dc91469a90ae127 P 819af79e262244839ea1ff6ed5aec230: Deleting WAL directory at /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/minicluster-data/ts-1/wal/wals/0266d73c65cb497e9dc91469a90ae127
I20260620 18:35:44.213734 24961 ts_tablet_manager.cc:1939] T 0266d73c65cb497e9dc91469a90ae127 P 819af79e262244839ea1ff6ed5aec230: Deleting consensus metadata
I20260620 18:35:44.213714 24833 tablet_service.cc:1467] Processing CreateTablet for tablet 4d9fb0c1a6404505bf633b7c521e662d (DEFAULT_TABLE table=test-table [id=2484c1324ccd4b64b6b8c90c2a4ad002]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:35:44.214097 24833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d9fb0c1a6404505bf633b7c521e662d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:35:44.215600 24398 catalog_manager.cc:4841] TS 819af79e262244839ea1ff6ed5aec230 (127.22.37.2:33193): tablet 0266d73c65cb497e9dc91469a90ae127 (table test-table [id=9228f399216e4435b48f1b9e8df886b1]) successfully deleted
I20260620 18:35:44.216678 24918 tablet_bootstrap.cc:492] T 4d9fb0c1a6404505bf633b7c521e662d P 819af79e262244839ea1ff6ed5aec230: Bootstrap starting.
I20260620 18:35:44.218302 24918 tablet_bootstrap.cc:654] T 4d9fb0c1a6404505bf633b7c521e662d P 819af79e262244839ea1ff6ed5aec230: Neither blocks nor log segments found. Creating new log.
I20260620 18:35:44.220983 24922 tablet_bootstrap.cc:492] T 4d9fb0c1a6404505bf633b7c521e662d P a6facfe4b14c490e8cc97f3d40b6d4df: Bootstrap starting.
I20260620 18:35:44.223105 24922 tablet_bootstrap.cc:654] T 4d9fb0c1a6404505bf633b7c521e662d P a6facfe4b14c490e8cc97f3d40b6d4df: Neither blocks nor log segments found. Creating new log.
I20260620 18:35:44.223925 24919 tablet_bootstrap.cc:492] T 4d9fb0c1a6404505bf633b7c521e662d P f5ec1f95ec4544deb9e4ba19316d3e02: Bootstrap starting.
I20260620 18:35:44.224762 24962 ts_tablet_manager.cc:1905] T 0266d73c65cb497e9dc91469a90ae127 P f5ec1f95ec4544deb9e4ba19316d3e02: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:35:44.225736 24919 tablet_bootstrap.cc:654] T 4d9fb0c1a6404505bf633b7c521e662d P f5ec1f95ec4544deb9e4ba19316d3e02: Neither blocks nor log segments found. Creating new log.
I20260620 18:35:44.231588 24919 tablet_bootstrap.cc:492] T 4d9fb0c1a6404505bf633b7c521e662d P f5ec1f95ec4544deb9e4ba19316d3e02: No bootstrap required, opened a new log
I20260620 18:35:44.231799 24919 ts_tablet_manager.cc:1397] T 4d9fb0c1a6404505bf633b7c521e662d P f5ec1f95ec4544deb9e4ba19316d3e02: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.002s
I20260620 18:35:44.232579 24919 raft_consensus.cc:348] T 4d9fb0c1a6404505bf633b7c521e662d P f5ec1f95ec4544deb9e4ba19316d3e02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ec1f95ec4544deb9e4ba19316d3e02" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 39519 } } peers { permanent_uuid: "819af79e262244839ea1ff6ed5aec230" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 33193 } } peers { permanent_uuid: "a6facfe4b14c490e8cc97f3d40b6d4df" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35601 } }
I20260620 18:35:44.232847 24919 raft_consensus.cc:374] T 4d9fb0c1a6404505bf633b7c521e662d P f5ec1f95ec4544deb9e4ba19316d3e02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:35:44.232942 24919 raft_consensus.cc:729] T 4d9fb0c1a6404505bf633b7c521e662d P f5ec1f95ec4544deb9e4ba19316d3e02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5ec1f95ec4544deb9e4ba19316d3e02, State: Initialized, Role: FOLLOWER
I20260620 18:35:44.233337 24919 consensus_queue.cc:260] T 4d9fb0c1a6404505bf633b7c521e662d P f5ec1f95ec4544deb9e4ba19316d3e02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ec1f95ec4544deb9e4ba19316d3e02" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 39519 } } peers { permanent_uuid: "819af79e262244839ea1ff6ed5aec230" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 33193 } } peers { permanent_uuid: "a6facfe4b14c490e8cc97f3d40b6d4df" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35601 } }
I20260620 18:35:44.233749 24918 tablet_bootstrap.cc:492] T 4d9fb0c1a6404505bf633b7c521e662d P 819af79e262244839ea1ff6ed5aec230: No bootstrap required, opened a new log
I20260620 18:35:44.233940 24918 ts_tablet_manager.cc:1397] T 4d9fb0c1a6404505bf633b7c521e662d P 819af79e262244839ea1ff6ed5aec230: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.005s
I20260620 18:35:44.234018 24919 ts_tablet_manager.cc:1428] T 4d9fb0c1a6404505bf633b7c521e662d P f5ec1f95ec4544deb9e4ba19316d3e02: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 18:35:44.234699 24918 raft_consensus.cc:348] T 4d9fb0c1a6404505bf633b7c521e662d P 819af79e262244839ea1ff6ed5aec230 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ec1f95ec4544deb9e4ba19316d3e02" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 39519 } } peers { permanent_uuid: "819af79e262244839ea1ff6ed5aec230" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 33193 } } peers { permanent_uuid: "a6facfe4b14c490e8cc97f3d40b6d4df" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35601 } }
I20260620 18:35:44.234954 24918 raft_consensus.cc:374] T 4d9fb0c1a6404505bf633b7c521e662d P 819af79e262244839ea1ff6ed5aec230 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:35:44.235023 24918 raft_consensus.cc:729] T 4d9fb0c1a6404505bf633b7c521e662d P 819af79e262244839ea1ff6ed5aec230 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819af79e262244839ea1ff6ed5aec230, State: Initialized, Role: FOLLOWER
I20260620 18:35:44.235260 24918 consensus_queue.cc:260] T 4d9fb0c1a6404505bf633b7c521e662d P 819af79e262244839ea1ff6ed5aec230 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ec1f95ec4544deb9e4ba19316d3e02" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 39519 } } peers { permanent_uuid: "819af79e262244839ea1ff6ed5aec230" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 33193 } } peers { permanent_uuid: "a6facfe4b14c490e8cc97f3d40b6d4df" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35601 } }
I20260620 18:35:44.236063 24962 ts_tablet_manager.cc:1918] T 0266d73c65cb497e9dc91469a90ae127 P f5ec1f95ec4544deb9e4ba19316d3e02: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:35:44.236230 24962 log.cc:1196] T 0266d73c65cb497e9dc91469a90ae127 P f5ec1f95ec4544deb9e4ba19316d3e02: Deleting WAL directory at /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/minicluster-data/ts-0/wal/wals/0266d73c65cb497e9dc91469a90ae127
I20260620 18:35:44.236824 24962 ts_tablet_manager.cc:1939] T 0266d73c65cb497e9dc91469a90ae127 P f5ec1f95ec4544deb9e4ba19316d3e02: Deleting consensus metadata
I20260620 18:35:44.236796 24963 ts_tablet_manager.cc:1918] T 0266d73c65cb497e9dc91469a90ae127 P a6facfe4b14c490e8cc97f3d40b6d4df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:35:44.236997 24963 log.cc:1196] T 0266d73c65cb497e9dc91469a90ae127 P a6facfe4b14c490e8cc97f3d40b6d4df: Deleting WAL directory at /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/minicluster-data/ts-2/wal/wals/0266d73c65cb497e9dc91469a90ae127
I20260620 18:35:44.237979 24963 ts_tablet_manager.cc:1939] T 0266d73c65cb497e9dc91469a90ae127 P a6facfe4b14c490e8cc97f3d40b6d4df: Deleting consensus metadata
I20260620 18:35:44.238590 24399 catalog_manager.cc:4841] TS f5ec1f95ec4544deb9e4ba19316d3e02 (127.22.37.1:39519): tablet 0266d73c65cb497e9dc91469a90ae127 (table test-table [id=9228f399216e4435b48f1b9e8df886b1]) successfully deleted
I20260620 18:35:44.239383 24918 ts_tablet_manager.cc:1428] T 4d9fb0c1a6404505bf633b7c521e662d P 819af79e262244839ea1ff6ed5aec230: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20260620 18:35:44.240250 24922 tablet_bootstrap.cc:492] T 4d9fb0c1a6404505bf633b7c521e662d P a6facfe4b14c490e8cc97f3d40b6d4df: No bootstrap required, opened a new log
I20260620 18:35:44.240411 24922 ts_tablet_manager.cc:1397] T 4d9fb0c1a6404505bf633b7c521e662d P a6facfe4b14c490e8cc97f3d40b6d4df: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.006s
I20260620 18:35:44.240479 24399 catalog_manager.cc:4841] TS a6facfe4b14c490e8cc97f3d40b6d4df (127.22.37.3:35601): tablet 0266d73c65cb497e9dc91469a90ae127 (table test-table [id=9228f399216e4435b48f1b9e8df886b1]) successfully deleted
I20260620 18:35:44.241413 24922 raft_consensus.cc:348] T 4d9fb0c1a6404505bf633b7c521e662d P a6facfe4b14c490e8cc97f3d40b6d4df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ec1f95ec4544deb9e4ba19316d3e02" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 39519 } } peers { permanent_uuid: "819af79e262244839ea1ff6ed5aec230" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 33193 } } peers { permanent_uuid: "a6facfe4b14c490e8cc97f3d40b6d4df" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35601 } }
I20260620 18:35:44.241641 24922 raft_consensus.cc:374] T 4d9fb0c1a6404505bf633b7c521e662d P a6facfe4b14c490e8cc97f3d40b6d4df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:35:44.241750 24922 raft_consensus.cc:729] T 4d9fb0c1a6404505bf633b7c521e662d P a6facfe4b14c490e8cc97f3d40b6d4df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6facfe4b14c490e8cc97f3d40b6d4df, State: Initialized, Role: FOLLOWER
I20260620 18:35:44.241966 24922 consensus_queue.cc:260] T 4d9fb0c1a6404505bf633b7c521e662d P a6facfe4b14c490e8cc97f3d40b6d4df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ec1f95ec4544deb9e4ba19316d3e02" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 39519 } } peers { permanent_uuid: "819af79e262244839ea1ff6ed5aec230" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 33193 } } peers { permanent_uuid: "a6facfe4b14c490e8cc97f3d40b6d4df" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35601 } }
I20260620 18:35:44.242453 24922 ts_tablet_manager.cc:1428] T 4d9fb0c1a6404505bf633b7c521e662d P a6facfe4b14c490e8cc97f3d40b6d4df: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 18:35:44.372309 24957 raft_consensus.cc:482] T 4d9fb0c1a6404505bf633b7c521e662d P f5ec1f95ec4544deb9e4ba19316d3e02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:35:44.372592 24957 raft_consensus.cc:504] T 4d9fb0c1a6404505bf633b7c521e662d P f5ec1f95ec4544deb9e4ba19316d3e02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ec1f95ec4544deb9e4ba19316d3e02" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 39519 } } peers { permanent_uuid: "819af79e262244839ea1ff6ed5aec230" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 33193 } } peers { permanent_uuid: "a6facfe4b14c490e8cc97f3d40b6d4df" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35601 } }
I20260620 18:35:44.373497 24957 leader_election.cc:288] T 4d9fb0c1a6404505bf633b7c521e662d P f5ec1f95ec4544deb9e4ba19316d3e02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 819af79e262244839ea1ff6ed5aec230 (127.22.37.2:33193), a6facfe4b14c490e8cc97f3d40b6d4df (127.22.37.3:35601)
I20260620 18:35:44.374164 24854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d9fb0c1a6404505bf633b7c521e662d" candidate_uuid: "f5ec1f95ec4544deb9e4ba19316d3e02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6facfe4b14c490e8cc97f3d40b6d4df" is_pre_election: true
I20260620 18:35:44.374112 24714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d9fb0c1a6404505bf633b7c521e662d" candidate_uuid: "f5ec1f95ec4544deb9e4ba19316d3e02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819af79e262244839ea1ff6ed5aec230" is_pre_election: true
I20260620 18:35:44.374426 24854 raft_consensus.cc:2454] T 4d9fb0c1a6404505bf633b7c521e662d P a6facfe4b14c490e8cc97f3d40b6d4df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5ec1f95ec4544deb9e4ba19316d3e02 in term 0.
I20260620 18:35:44.374444 24714 raft_consensus.cc:2454] T 4d9fb0c1a6404505bf633b7c521e662d P 819af79e262244839ea1ff6ed5aec230 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5ec1f95ec4544deb9e4ba19316d3e02 in term 0.
I20260620 18:35:44.374918 24502 leader_election.cc:302] T 4d9fb0c1a6404505bf633b7c521e662d P f5ec1f95ec4544deb9e4ba19316d3e02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6facfe4b14c490e8cc97f3d40b6d4df, f5ec1f95ec4544deb9e4ba19316d3e02; no voters:
I20260620 18:35:44.375273 24957 raft_consensus.cc:2781] T 4d9fb0c1a6404505bf633b7c521e662d P f5ec1f95ec4544deb9e4ba19316d3e02 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:35:44.375412 24957 raft_consensus.cc:482] T 4d9fb0c1a6404505bf633b7c521e662d P f5ec1f95ec4544deb9e4ba19316d3e02 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:35:44.375519 24957 raft_consensus.cc:3037] T 4d9fb0c1a6404505bf633b7c521e662d P f5ec1f95ec4544deb9e4ba19316d3e02 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:35:44.378916 24957 raft_consensus.cc:504] T 4d9fb0c1a6404505bf633b7c521e662d P f5ec1f95ec4544deb9e4ba19316d3e02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ec1f95ec4544deb9e4ba19316d3e02" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 39519 } } peers { permanent_uuid: "819af79e262244839ea1ff6ed5aec230" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 33193 } } peers { permanent_uuid: "a6facfe4b14c490e8cc97f3d40b6d4df" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35601 } }
I20260620 18:35:44.379638 24957 leader_election.cc:288] T 4d9fb0c1a6404505bf633b7c521e662d P f5ec1f95ec4544deb9e4ba19316d3e02 [CANDIDATE]: Term 1 election: Requested vote from peers 819af79e262244839ea1ff6ed5aec230 (127.22.37.2:33193), a6facfe4b14c490e8cc97f3d40b6d4df (127.22.37.3:35601)
I20260620 18:35:44.380155 24854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d9fb0c1a6404505bf633b7c521e662d" candidate_uuid: "f5ec1f95ec4544deb9e4ba19316d3e02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6facfe4b14c490e8cc97f3d40b6d4df"
I20260620 18:35:44.380133 24714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d9fb0c1a6404505bf633b7c521e662d" candidate_uuid: "f5ec1f95ec4544deb9e4ba19316d3e02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819af79e262244839ea1ff6ed5aec230"
I20260620 18:35:44.380360 24854 raft_consensus.cc:3037] T 4d9fb0c1a6404505bf633b7c521e662d P a6facfe4b14c490e8cc97f3d40b6d4df [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:35:44.380378 24714 raft_consensus.cc:3037] T 4d9fb0c1a6404505bf633b7c521e662d P 819af79e262244839ea1ff6ed5aec230 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:35:44.382797 24854 raft_consensus.cc:2454] T 4d9fb0c1a6404505bf633b7c521e662d P a6facfe4b14c490e8cc97f3d40b6d4df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5ec1f95ec4544deb9e4ba19316d3e02 in term 1.
I20260620 18:35:44.383258 24502 leader_election.cc:302] T 4d9fb0c1a6404505bf633b7c521e662d P f5ec1f95ec4544deb9e4ba19316d3e02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6facfe4b14c490e8cc97f3d40b6d4df, f5ec1f95ec4544deb9e4ba19316d3e02; no voters:
I20260620 18:35:44.383462 24714 raft_consensus.cc:2454] T 4d9fb0c1a6404505bf633b7c521e662d P 819af79e262244839ea1ff6ed5aec230 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5ec1f95ec4544deb9e4ba19316d3e02 in term 1.
I20260620 18:35:44.383606 24957 raft_consensus.cc:2781] T 4d9fb0c1a6404505bf633b7c521e662d P f5ec1f95ec4544deb9e4ba19316d3e02 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:35:44.383858 24957 raft_consensus.cc:686] T 4d9fb0c1a6404505bf633b7c521e662d P f5ec1f95ec4544deb9e4ba19316d3e02 [term 1 LEADER]: Becoming Leader. State: Replica: f5ec1f95ec4544deb9e4ba19316d3e02, State: Running, Role: LEADER
I20260620 18:35:44.384140 24957 consensus_queue.cc:237] T 4d9fb0c1a6404505bf633b7c521e662d P f5ec1f95ec4544deb9e4ba19316d3e02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ec1f95ec4544deb9e4ba19316d3e02" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 39519 } } peers { permanent_uuid: "819af79e262244839ea1ff6ed5aec230" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 33193 } } peers { permanent_uuid: "a6facfe4b14c490e8cc97f3d40b6d4df" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35601 } }
I20260620 18:35:44.387064 24412 catalog_manager.cc:5495] T 4d9fb0c1a6404505bf633b7c521e662d P f5ec1f95ec4544deb9e4ba19316d3e02 reported cstate change: term changed from 0 to 1, leader changed from <none> to f5ec1f95ec4544deb9e4ba19316d3e02 (127.22.37.1). New cstate: current_term: 1 leader_uuid: "f5ec1f95ec4544deb9e4ba19316d3e02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ec1f95ec4544deb9e4ba19316d3e02" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 39519 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "819af79e262244839ea1ff6ed5aec230" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 33193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6facfe4b14c490e8cc97f3d40b6d4df" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35601 } health_report { overall_health: UNKNOWN } } }
I20260620 18:35:44.435868 24854 raft_consensus.cc:1261] T 4d9fb0c1a6404505bf633b7c521e662d P a6facfe4b14c490e8cc97f3d40b6d4df [term 1 FOLLOWER]: Refusing update from remote peer f5ec1f95ec4544deb9e4ba19316d3e02: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:35:44.436021 24714 raft_consensus.cc:1261] T 4d9fb0c1a6404505bf633b7c521e662d P 819af79e262244839ea1ff6ed5aec230 [term 1 FOLLOWER]: Refusing update from remote peer f5ec1f95ec4544deb9e4ba19316d3e02: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:35:44.436532 24957 consensus_queue.cc:1035] T 4d9fb0c1a6404505bf633b7c521e662d P f5ec1f95ec4544deb9e4ba19316d3e02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6facfe4b14c490e8cc97f3d40b6d4df" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:35:44.436882 24925 consensus_queue.cc:1035] T 4d9fb0c1a6404505bf633b7c521e662d P f5ec1f95ec4544deb9e4ba19316d3e02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "819af79e262244839ea1ff6ed5aec230" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 18:35:44.466171 24944 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 18:35:46.059463 24412 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39058:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:35:46.059724 24412 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39058:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:35:46.063683 24412 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 518479382dc0411d87ae9ad184d6250a: Sending DeleteTablet for 3 replicas of tablet 4d9fb0c1a6404505bf633b7c521e662d
I20260620 18:35:46.064613 24690 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d9fb0c1a6404505bf633b7c521e662d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:35:46 UTC) from {username='kudu', principal='kudu/127.22.37.62@KRBTEST.COM'} at 127.0.0.1:57084
I20260620 18:35:46.064615 24550 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d9fb0c1a6404505bf633b7c521e662d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:35:46 UTC) from {username='kudu', principal='kudu/127.22.37.62@KRBTEST.COM'} at 127.0.0.1:54838
I20260620 18:35:46.064649 24833 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d9fb0c1a6404505bf633b7c521e662d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:35:46 UTC) from {username='kudu', principal='kudu/127.22.37.62@KRBTEST.COM'} at 127.0.0.1:60390
I20260620 18:35:46.065358 24991 tablet_replica.cc:330] T 4d9fb0c1a6404505bf633b7c521e662d P a6facfe4b14c490e8cc97f3d40b6d4df: stopping tablet replica
I20260620 18:35:46.065728 24991 raft_consensus.cc:2229] T 4d9fb0c1a6404505bf633b7c521e662d P a6facfe4b14c490e8cc97f3d40b6d4df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:35:46.066090 24991 raft_consensus.cc:2258] T 4d9fb0c1a6404505bf633b7c521e662d P a6facfe4b14c490e8cc97f3d40b6d4df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:35:46.067462 24991 ts_tablet_manager.cc:1905] T 4d9fb0c1a6404505bf633b7c521e662d P a6facfe4b14c490e8cc97f3d40b6d4df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:35:46.076764 24989 tablet_replica.cc:330] T 4d9fb0c1a6404505bf633b7c521e662d P f5ec1f95ec4544deb9e4ba19316d3e02: stopping tablet replica
I20260620 18:35:46.077162 24989 raft_consensus.cc:2229] T 4d9fb0c1a6404505bf633b7c521e662d P f5ec1f95ec4544deb9e4ba19316d3e02 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:35:46.077307 24991 ts_tablet_manager.cc:1918] T 4d9fb0c1a6404505bf633b7c521e662d P a6facfe4b14c490e8cc97f3d40b6d4df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 18:35:46.077503 24991 log.cc:1196] T 4d9fb0c1a6404505bf633b7c521e662d P a6facfe4b14c490e8cc97f3d40b6d4df: Deleting WAL directory at /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/minicluster-data/ts-2/wal/wals/4d9fb0c1a6404505bf633b7c521e662d
I20260620 18:35:46.077764 24989 raft_consensus.cc:2258] T 4d9fb0c1a6404505bf633b7c521e662d P f5ec1f95ec4544deb9e4ba19316d3e02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:35:46.078047 24991 ts_tablet_manager.cc:1939] T 4d9fb0c1a6404505bf633b7c521e662d P a6facfe4b14c490e8cc97f3d40b6d4df: Deleting consensus metadata
I20260620 18:35:46.078392 24990 tablet_replica.cc:330] T 4d9fb0c1a6404505bf633b7c521e662d P 819af79e262244839ea1ff6ed5aec230: stopping tablet replica
I20260620 18:35:46.078626 24990 raft_consensus.cc:2229] T 4d9fb0c1a6404505bf633b7c521e662d P 819af79e262244839ea1ff6ed5aec230 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:35:46.078900 24990 raft_consensus.cc:2258] T 4d9fb0c1a6404505bf633b7c521e662d P 819af79e262244839ea1ff6ed5aec230 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:35:46.079103 24989 ts_tablet_manager.cc:1905] T 4d9fb0c1a6404505bf633b7c521e662d P f5ec1f95ec4544deb9e4ba19316d3e02: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:35:46.079341 24399 catalog_manager.cc:4841] TS a6facfe4b14c490e8cc97f3d40b6d4df (127.22.37.3:35601): tablet 4d9fb0c1a6404505bf633b7c521e662d (table test-table [id=2484c1324ccd4b64b6b8c90c2a4ad002]) successfully deleted
I20260620 18:35:46.079882 24990 ts_tablet_manager.cc:1905] T 4d9fb0c1a6404505bf633b7c521e662d P 819af79e262244839ea1ff6ed5aec230: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:35:46.084643 24990 ts_tablet_manager.cc:1918] T 4d9fb0c1a6404505bf633b7c521e662d P 819af79e262244839ea1ff6ed5aec230: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 18:35:46.084787 24990 log.cc:1196] T 4d9fb0c1a6404505bf633b7c521e662d P 819af79e262244839ea1ff6ed5aec230: Deleting WAL directory at /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/minicluster-data/ts-1/wal/wals/4d9fb0c1a6404505bf633b7c521e662d
I20260620 18:35:46.084884 24989 ts_tablet_manager.cc:1918] T 4d9fb0c1a6404505bf633b7c521e662d P f5ec1f95ec4544deb9e4ba19316d3e02: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 18:35:46.085029 24989 log.cc:1196] T 4d9fb0c1a6404505bf633b7c521e662d P f5ec1f95ec4544deb9e4ba19316d3e02: Deleting WAL directory at /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/minicluster-data/ts-0/wal/wals/4d9fb0c1a6404505bf633b7c521e662d
I20260620 18:35:46.085282 24990 ts_tablet_manager.cc:1939] T 4d9fb0c1a6404505bf633b7c521e662d P 819af79e262244839ea1ff6ed5aec230: Deleting consensus metadata
I20260620 18:35:46.085549 24989 ts_tablet_manager.cc:1939] T 4d9fb0c1a6404505bf633b7c521e662d P f5ec1f95ec4544deb9e4ba19316d3e02: Deleting consensus metadata
I20260620 18:35:46.086431 24398 catalog_manager.cc:4841] TS 819af79e262244839ea1ff6ed5aec230 (127.22.37.2:33193): tablet 4d9fb0c1a6404505bf633b7c521e662d (table test-table [id=2484c1324ccd4b64b6b8c90c2a4ad002]) successfully deleted
I20260620 18:35:46.086619 24399 catalog_manager.cc:4841] TS f5ec1f95ec4544deb9e4ba19316d3e02 (127.22.37.1:39519): tablet 4d9fb0c1a6404505bf633b7c521e662d (table test-table [id=2484c1324ccd4b64b6b8c90c2a4ad002]) successfully deleted
Jun 20 18:35:46 dist-test-slave-5lgp krb5kdc[24369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980546, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 18:35:46 dist-test-slave-5lgp krb5kdc[24369](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980546, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM
I20260620 18:35:46.130290 24999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:46.112366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39066 (local address 127.22.37.62:38817)
0620 18:35:46.112914 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:46.112918 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:46.112942 (+ 24us) server_negotiation.cc:407] Connection header received
0620 18:35:46.113123 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:46.113127 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:46.113196 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:46.113286 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:46.114575 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:46.115311 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:46.116495 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:46.116874 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:46.116964 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:46.125501 (+ 8537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:46.125507 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:46.125510 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:46.125545 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:46.127928 (+ 2383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:46.128509 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:46.128513 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:46.128516 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:46.128624 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:46.128931 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:46.128934 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:46.128936 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:46.129184 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:46.129481 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:46.129718 (+ 237us) server_negotiation.cc:299] Negotiation successful
0620 18:35:46.129924 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":281,"threads_started":1}
I20260620 18:35:46.133903 24412 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39066:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:35:46.134428 24412 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:35:46.140834 24550 tablet_service.cc:1467] Processing CreateTablet for tablet a194f47e7fc340209896001c3bf902d4 (DEFAULT_TABLE table=test-table [id=99db8a67d95742499d5748a8d2ce933d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:35:46.140923 24690 tablet_service.cc:1467] Processing CreateTablet for tablet a194f47e7fc340209896001c3bf902d4 (DEFAULT_TABLE table=test-table [id=99db8a67d95742499d5748a8d2ce933d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:35:46.141300 24550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a194f47e7fc340209896001c3bf902d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:35:46.141356 24690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a194f47e7fc340209896001c3bf902d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:35:46.141352 24833 tablet_service.cc:1467] Processing CreateTablet for tablet a194f47e7fc340209896001c3bf902d4 (DEFAULT_TABLE table=test-table [id=99db8a67d95742499d5748a8d2ce933d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:35:46.141780 24833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a194f47e7fc340209896001c3bf902d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:35:46.147770 25000 tablet_bootstrap.cc:492] T a194f47e7fc340209896001c3bf902d4 P f5ec1f95ec4544deb9e4ba19316d3e02: Bootstrap starting.
I20260620 18:35:46.148906 25002 tablet_bootstrap.cc:492] T a194f47e7fc340209896001c3bf902d4 P a6facfe4b14c490e8cc97f3d40b6d4df: Bootstrap starting.
I20260620 18:35:46.149143 25001 tablet_bootstrap.cc:492] T a194f47e7fc340209896001c3bf902d4 P 819af79e262244839ea1ff6ed5aec230: Bootstrap starting.
I20260620 18:35:46.150070 25000 tablet_bootstrap.cc:654] T a194f47e7fc340209896001c3bf902d4 P f5ec1f95ec4544deb9e4ba19316d3e02: Neither blocks nor log segments found. Creating new log.
I20260620 18:35:46.150666 25002 tablet_bootstrap.cc:654] T a194f47e7fc340209896001c3bf902d4 P a6facfe4b14c490e8cc97f3d40b6d4df: Neither blocks nor log segments found. Creating new log.
I20260620 18:35:46.150889 25001 tablet_bootstrap.cc:654] T a194f47e7fc340209896001c3bf902d4 P 819af79e262244839ea1ff6ed5aec230: Neither blocks nor log segments found. Creating new log.
I20260620 18:35:46.152752 25000 tablet_bootstrap.cc:492] T a194f47e7fc340209896001c3bf902d4 P f5ec1f95ec4544deb9e4ba19316d3e02: No bootstrap required, opened a new log
I20260620 18:35:46.152904 25000 ts_tablet_manager.cc:1397] T a194f47e7fc340209896001c3bf902d4 P f5ec1f95ec4544deb9e4ba19316d3e02: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 18:35:46.153287 25002 tablet_bootstrap.cc:492] T a194f47e7fc340209896001c3bf902d4 P a6facfe4b14c490e8cc97f3d40b6d4df: No bootstrap required, opened a new log
I20260620 18:35:46.153450 25001 tablet_bootstrap.cc:492] T a194f47e7fc340209896001c3bf902d4 P 819af79e262244839ea1ff6ed5aec230: No bootstrap required, opened a new log
I20260620 18:35:46.153453 25002 ts_tablet_manager.cc:1397] T a194f47e7fc340209896001c3bf902d4 P a6facfe4b14c490e8cc97f3d40b6d4df: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 18:35:46.153610 25001 ts_tablet_manager.cc:1397] T a194f47e7fc340209896001c3bf902d4 P 819af79e262244839ea1ff6ed5aec230: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260620 18:35:46.153779 25000 raft_consensus.cc:348] T a194f47e7fc340209896001c3bf902d4 P f5ec1f95ec4544deb9e4ba19316d3e02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ec1f95ec4544deb9e4ba19316d3e02" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 39519 } } peers { permanent_uuid: "819af79e262244839ea1ff6ed5aec230" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 33193 } } peers { permanent_uuid: "a6facfe4b14c490e8cc97f3d40b6d4df" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35601 } }
I20260620 18:35:46.154036 25000 raft_consensus.cc:374] T a194f47e7fc340209896001c3bf902d4 P f5ec1f95ec4544deb9e4ba19316d3e02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:35:46.154122 25000 raft_consensus.cc:729] T a194f47e7fc340209896001c3bf902d4 P f5ec1f95ec4544deb9e4ba19316d3e02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5ec1f95ec4544deb9e4ba19316d3e02, State: Initialized, Role: FOLLOWER
I20260620 18:35:46.154306 25002 raft_consensus.cc:348] T a194f47e7fc340209896001c3bf902d4 P a6facfe4b14c490e8cc97f3d40b6d4df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ec1f95ec4544deb9e4ba19316d3e02" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 39519 } } peers { permanent_uuid: "819af79e262244839ea1ff6ed5aec230" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 33193 } } peers { permanent_uuid: "a6facfe4b14c490e8cc97f3d40b6d4df" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35601 } }
I20260620 18:35:46.154356 25000 consensus_queue.cc:260] T a194f47e7fc340209896001c3bf902d4 P f5ec1f95ec4544deb9e4ba19316d3e02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ec1f95ec4544deb9e4ba19316d3e02" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 39519 } } peers { permanent_uuid: "819af79e262244839ea1ff6ed5aec230" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 33193 } } peers { permanent_uuid: "a6facfe4b14c490e8cc97f3d40b6d4df" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35601 } }
I20260620 18:35:46.154547 25002 raft_consensus.cc:374] T a194f47e7fc340209896001c3bf902d4 P a6facfe4b14c490e8cc97f3d40b6d4df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:35:46.154425 25001 raft_consensus.cc:348] T a194f47e7fc340209896001c3bf902d4 P 819af79e262244839ea1ff6ed5aec230 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ec1f95ec4544deb9e4ba19316d3e02" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 39519 } } peers { permanent_uuid: "819af79e262244839ea1ff6ed5aec230" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 33193 } } peers { permanent_uuid: "a6facfe4b14c490e8cc97f3d40b6d4df" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35601 } }
I20260620 18:35:46.154630 25002 raft_consensus.cc:729] T a194f47e7fc340209896001c3bf902d4 P a6facfe4b14c490e8cc97f3d40b6d4df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6facfe4b14c490e8cc97f3d40b6d4df, State: Initialized, Role: FOLLOWER
I20260620 18:35:46.154672 25001 raft_consensus.cc:374] T a194f47e7fc340209896001c3bf902d4 P 819af79e262244839ea1ff6ed5aec230 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:35:46.154767 25001 raft_consensus.cc:729] T a194f47e7fc340209896001c3bf902d4 P 819af79e262244839ea1ff6ed5aec230 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819af79e262244839ea1ff6ed5aec230, State: Initialized, Role: FOLLOWER
I20260620 18:35:46.154918 25000 ts_tablet_manager.cc:1428] T a194f47e7fc340209896001c3bf902d4 P f5ec1f95ec4544deb9e4ba19316d3e02: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 18:35:46.154861 25002 consensus_queue.cc:260] T a194f47e7fc340209896001c3bf902d4 P a6facfe4b14c490e8cc97f3d40b6d4df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ec1f95ec4544deb9e4ba19316d3e02" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 39519 } } peers { permanent_uuid: "819af79e262244839ea1ff6ed5aec230" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 33193 } } peers { permanent_uuid: "a6facfe4b14c490e8cc97f3d40b6d4df" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35601 } }
I20260620 18:35:46.154997 25001 consensus_queue.cc:260] T a194f47e7fc340209896001c3bf902d4 P 819af79e262244839ea1ff6ed5aec230 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ec1f95ec4544deb9e4ba19316d3e02" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 39519 } } peers { permanent_uuid: "819af79e262244839ea1ff6ed5aec230" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 33193 } } peers { permanent_uuid: "a6facfe4b14c490e8cc97f3d40b6d4df" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35601 } }
I20260620 18:35:46.155897 25001 ts_tablet_manager.cc:1428] T a194f47e7fc340209896001c3bf902d4 P 819af79e262244839ea1ff6ed5aec230: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 18:35:46.157816 25002 ts_tablet_manager.cc:1428] T a194f47e7fc340209896001c3bf902d4 P a6facfe4b14c490e8cc97f3d40b6d4df: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260620 18:35:46.361250 24972 raft_consensus.cc:482] T a194f47e7fc340209896001c3bf902d4 P f5ec1f95ec4544deb9e4ba19316d3e02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:35:46.361575 24972 raft_consensus.cc:504] T a194f47e7fc340209896001c3bf902d4 P f5ec1f95ec4544deb9e4ba19316d3e02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ec1f95ec4544deb9e4ba19316d3e02" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 39519 } } peers { permanent_uuid: "819af79e262244839ea1ff6ed5aec230" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 33193 } } peers { permanent_uuid: "a6facfe4b14c490e8cc97f3d40b6d4df" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35601 } }
I20260620 18:35:46.362285 24972 leader_election.cc:288] T a194f47e7fc340209896001c3bf902d4 P f5ec1f95ec4544deb9e4ba19316d3e02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 819af79e262244839ea1ff6ed5aec230 (127.22.37.2:33193), a6facfe4b14c490e8cc97f3d40b6d4df (127.22.37.3:35601)
I20260620 18:35:46.362900 24853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a194f47e7fc340209896001c3bf902d4" candidate_uuid: "f5ec1f95ec4544deb9e4ba19316d3e02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6facfe4b14c490e8cc97f3d40b6d4df" is_pre_election: true
I20260620 18:35:46.362897 24714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a194f47e7fc340209896001c3bf902d4" candidate_uuid: "f5ec1f95ec4544deb9e4ba19316d3e02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819af79e262244839ea1ff6ed5aec230" is_pre_election: true
I20260620 18:35:46.363135 24853 raft_consensus.cc:2454] T a194f47e7fc340209896001c3bf902d4 P a6facfe4b14c490e8cc97f3d40b6d4df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5ec1f95ec4544deb9e4ba19316d3e02 in term 0.
I20260620 18:35:46.363178 24714 raft_consensus.cc:2454] T a194f47e7fc340209896001c3bf902d4 P 819af79e262244839ea1ff6ed5aec230 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5ec1f95ec4544deb9e4ba19316d3e02 in term 0.
I20260620 18:35:46.363687 24502 leader_election.cc:302] T a194f47e7fc340209896001c3bf902d4 P f5ec1f95ec4544deb9e4ba19316d3e02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6facfe4b14c490e8cc97f3d40b6d4df, f5ec1f95ec4544deb9e4ba19316d3e02; no voters:
I20260620 18:35:46.363942 24972 raft_consensus.cc:2781] T a194f47e7fc340209896001c3bf902d4 P f5ec1f95ec4544deb9e4ba19316d3e02 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:35:46.364035 24972 raft_consensus.cc:482] T a194f47e7fc340209896001c3bf902d4 P f5ec1f95ec4544deb9e4ba19316d3e02 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:35:46.364167 24972 raft_consensus.cc:3037] T a194f47e7fc340209896001c3bf902d4 P f5ec1f95ec4544deb9e4ba19316d3e02 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:35:46.366353 24972 raft_consensus.cc:504] T a194f47e7fc340209896001c3bf902d4 P f5ec1f95ec4544deb9e4ba19316d3e02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ec1f95ec4544deb9e4ba19316d3e02" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 39519 } } peers { permanent_uuid: "819af79e262244839ea1ff6ed5aec230" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 33193 } } peers { permanent_uuid: "a6facfe4b14c490e8cc97f3d40b6d4df" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35601 } }
I20260620 18:35:46.366842 24972 leader_election.cc:288] T a194f47e7fc340209896001c3bf902d4 P f5ec1f95ec4544deb9e4ba19316d3e02 [CANDIDATE]: Term 1 election: Requested vote from peers 819af79e262244839ea1ff6ed5aec230 (127.22.37.2:33193), a6facfe4b14c490e8cc97f3d40b6d4df (127.22.37.3:35601)
I20260620 18:35:46.367372 24853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a194f47e7fc340209896001c3bf902d4" candidate_uuid: "f5ec1f95ec4544deb9e4ba19316d3e02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6facfe4b14c490e8cc97f3d40b6d4df"
I20260620 18:35:46.367370 24714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a194f47e7fc340209896001c3bf902d4" candidate_uuid: "f5ec1f95ec4544deb9e4ba19316d3e02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819af79e262244839ea1ff6ed5aec230"
I20260620 18:35:46.367560 24853 raft_consensus.cc:3037] T a194f47e7fc340209896001c3bf902d4 P a6facfe4b14c490e8cc97f3d40b6d4df [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:35:46.367607 24714 raft_consensus.cc:3037] T a194f47e7fc340209896001c3bf902d4 P 819af79e262244839ea1ff6ed5aec230 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:35:46.369796 24714 raft_consensus.cc:2454] T a194f47e7fc340209896001c3bf902d4 P 819af79e262244839ea1ff6ed5aec230 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5ec1f95ec4544deb9e4ba19316d3e02 in term 1.
I20260620 18:35:46.369987 24853 raft_consensus.cc:2454] T a194f47e7fc340209896001c3bf902d4 P a6facfe4b14c490e8cc97f3d40b6d4df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5ec1f95ec4544deb9e4ba19316d3e02 in term 1.
I20260620 18:35:46.370227 24501 leader_election.cc:302] T a194f47e7fc340209896001c3bf902d4 P f5ec1f95ec4544deb9e4ba19316d3e02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 819af79e262244839ea1ff6ed5aec230, f5ec1f95ec4544deb9e4ba19316d3e02; no voters:
I20260620 18:35:46.370540 24972 raft_consensus.cc:2781] T a194f47e7fc340209896001c3bf902d4 P f5ec1f95ec4544deb9e4ba19316d3e02 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:35:46.370688 24972 raft_consensus.cc:686] T a194f47e7fc340209896001c3bf902d4 P f5ec1f95ec4544deb9e4ba19316d3e02 [term 1 LEADER]: Becoming Leader. State: Replica: f5ec1f95ec4544deb9e4ba19316d3e02, State: Running, Role: LEADER
I20260620 18:35:46.370955 24972 consensus_queue.cc:237] T a194f47e7fc340209896001c3bf902d4 P f5ec1f95ec4544deb9e4ba19316d3e02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ec1f95ec4544deb9e4ba19316d3e02" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 39519 } } peers { permanent_uuid: "819af79e262244839ea1ff6ed5aec230" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 33193 } } peers { permanent_uuid: "a6facfe4b14c490e8cc97f3d40b6d4df" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35601 } }
I20260620 18:35:46.373647 24414 catalog_manager.cc:5495] T a194f47e7fc340209896001c3bf902d4 P f5ec1f95ec4544deb9e4ba19316d3e02 reported cstate change: term changed from 0 to 1, leader changed from <none> to f5ec1f95ec4544deb9e4ba19316d3e02 (127.22.37.1). New cstate: current_term: 1 leader_uuid: "f5ec1f95ec4544deb9e4ba19316d3e02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ec1f95ec4544deb9e4ba19316d3e02" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 39519 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "819af79e262244839ea1ff6ed5aec230" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 33193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6facfe4b14c490e8cc97f3d40b6d4df" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35601 } health_report { overall_health: UNKNOWN } } }
I20260620 18:35:46.397976 25010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:46.389785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54876 (local address 127.22.37.1:39519)
0620 18:35:46.393757 (+ 3972us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:46.393761 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:46.393779 (+ 18us) server_negotiation.cc:407] Connection header received
0620 18:35:46.393840 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:46.393843 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:46.393907 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:46.393994 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:35:46.394337 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:46.395131 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:46.396364 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:46.396767 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:46.396854 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:46.397225 (+ 371us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:35:46.397314 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:35:46.397403 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:46.397565 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 18:35:46.397621 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3875,"thread_start_us":3771,"threads_started":1}
I20260620 18:35:46.406175 24853 raft_consensus.cc:1261] T a194f47e7fc340209896001c3bf902d4 P a6facfe4b14c490e8cc97f3d40b6d4df [term 1 FOLLOWER]: Refusing update from remote peer f5ec1f95ec4544deb9e4ba19316d3e02: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:35:46.406270 24714 raft_consensus.cc:1261] T a194f47e7fc340209896001c3bf902d4 P 819af79e262244839ea1ff6ed5aec230 [term 1 FOLLOWER]: Refusing update from remote peer f5ec1f95ec4544deb9e4ba19316d3e02: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:35:46.406706 24929 consensus_queue.cc:1035] T a194f47e7fc340209896001c3bf902d4 P f5ec1f95ec4544deb9e4ba19316d3e02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6facfe4b14c490e8cc97f3d40b6d4df" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:35:46.407006 24978 consensus_queue.cc:1035] T a194f47e7fc340209896001c3bf902d4 P f5ec1f95ec4544deb9e4ba19316d3e02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "819af79e262244839ea1ff6ed5aec230" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:35:46.413906 24974 mvcc.cc:205] Tried to move back new op lower bound from 7298992318074945536 to 7298992317937348608. Current Snapshot: MvccSnapshot[applied={T|T < 7298992318074945536}]
I20260620 18:35:46.448688 24413 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39066:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:35:46.448951 24413 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39066:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:35:46.452562 24413 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 518479382dc0411d87ae9ad184d6250a: Sending DeleteTablet for 3 replicas of tablet a194f47e7fc340209896001c3bf902d4
I20260620 18:35:46.453269 24551 tablet_service.cc:1514] Processing DeleteTablet for tablet a194f47e7fc340209896001c3bf902d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:35:46 UTC) from {username='kudu', principal='kudu/127.22.37.62@KRBTEST.COM'} at 127.0.0.1:54838
I20260620 18:35:46.453353 24690 tablet_service.cc:1514] Processing DeleteTablet for tablet a194f47e7fc340209896001c3bf902d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:35:46 UTC) from {username='kudu', principal='kudu/127.22.37.62@KRBTEST.COM'} at 127.0.0.1:57084
I20260620 18:35:46.453609 24989 tablet_replica.cc:330] T a194f47e7fc340209896001c3bf902d4 P f5ec1f95ec4544deb9e4ba19316d3e02: stopping tablet replica
I20260620 18:35:46.453593 24833 tablet_service.cc:1514] Processing DeleteTablet for tablet a194f47e7fc340209896001c3bf902d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:35:46 UTC) from {username='kudu', principal='kudu/127.22.37.62@KRBTEST.COM'} at 127.0.0.1:60390
I20260620 18:35:46.453796 24989 raft_consensus.cc:2229] T a194f47e7fc340209896001c3bf902d4 P f5ec1f95ec4544deb9e4ba19316d3e02 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:35:46.453871 24990 tablet_replica.cc:330] T a194f47e7fc340209896001c3bf902d4 P 819af79e262244839ea1ff6ed5aec230: stopping tablet replica
I20260620 18:35:46.454049 24990 raft_consensus.cc:2229] T a194f47e7fc340209896001c3bf902d4 P 819af79e262244839ea1ff6ed5aec230 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:35:46.454077 24989 raft_consensus.cc:2258] T a194f47e7fc340209896001c3bf902d4 P f5ec1f95ec4544deb9e4ba19316d3e02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:35:46.454226 24990 raft_consensus.cc:2258] T a194f47e7fc340209896001c3bf902d4 P 819af79e262244839ea1ff6ed5aec230 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:35:46.454490 24991 tablet_replica.cc:330] T a194f47e7fc340209896001c3bf902d4 P a6facfe4b14c490e8cc97f3d40b6d4df: stopping tablet replica
I20260620 18:35:46.454694 24991 raft_consensus.cc:2229] T a194f47e7fc340209896001c3bf902d4 P a6facfe4b14c490e8cc97f3d40b6d4df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:35:46.454876 24991 raft_consensus.cc:2258] T a194f47e7fc340209896001c3bf902d4 P a6facfe4b14c490e8cc97f3d40b6d4df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:35:46.454963 24989 ts_tablet_manager.cc:1905] T a194f47e7fc340209896001c3bf902d4 P f5ec1f95ec4544deb9e4ba19316d3e02: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:35:46.455047 24413 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39066:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 18:35:46.455889 24991 ts_tablet_manager.cc:1905] T a194f47e7fc340209896001c3bf902d4 P a6facfe4b14c490e8cc97f3d40b6d4df: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 18:35:46.455870 24413 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:35:46.460793 24990 ts_tablet_manager.cc:1905] T a194f47e7fc340209896001c3bf902d4 P 819af79e262244839ea1ff6ed5aec230: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:35:46.461411 24989 ts_tablet_manager.cc:1918] T a194f47e7fc340209896001c3bf902d4 P f5ec1f95ec4544deb9e4ba19316d3e02: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:35:46.461556 24989 log.cc:1196] T a194f47e7fc340209896001c3bf902d4 P f5ec1f95ec4544deb9e4ba19316d3e02: Deleting WAL directory at /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/minicluster-data/ts-0/wal/wals/a194f47e7fc340209896001c3bf902d4
I20260620 18:35:46.461942 24989 ts_tablet_manager.cc:1939] T a194f47e7fc340209896001c3bf902d4 P f5ec1f95ec4544deb9e4ba19316d3e02: Deleting consensus metadata
I20260620 18:35:46.463189 24399 catalog_manager.cc:4841] TS f5ec1f95ec4544deb9e4ba19316d3e02 (127.22.37.1:39519): tablet a194f47e7fc340209896001c3bf902d4 (table test-table [id=99db8a67d95742499d5748a8d2ce933d]) successfully deleted
I20260620 18:35:46.466364 24990 ts_tablet_manager.cc:1918] T a194f47e7fc340209896001c3bf902d4 P 819af79e262244839ea1ff6ed5aec230: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:35:46.466535 24990 log.cc:1196] T a194f47e7fc340209896001c3bf902d4 P 819af79e262244839ea1ff6ed5aec230: Deleting WAL directory at /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/minicluster-data/ts-1/wal/wals/a194f47e7fc340209896001c3bf902d4
I20260620 18:35:46.466910 24990 ts_tablet_manager.cc:1939] T a194f47e7fc340209896001c3bf902d4 P 819af79e262244839ea1ff6ed5aec230: Deleting consensus metadata
I20260620 18:35:46.467846 24398 catalog_manager.cc:4841] TS 819af79e262244839ea1ff6ed5aec230 (127.22.37.2:33193): tablet a194f47e7fc340209896001c3bf902d4 (table test-table [id=99db8a67d95742499d5748a8d2ce933d]) successfully deleted
I20260620 18:35:46.471176 24551 tablet_service.cc:1467] Processing CreateTablet for tablet 51765401d74449af81d542e48fb45d43 (DEFAULT_TABLE table=test-table [id=ea574c8b265d481ea956c18eb6309acb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:35:46.471629 24551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51765401d74449af81d542e48fb45d43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:35:46.471611 24690 tablet_service.cc:1467] Processing CreateTablet for tablet 51765401d74449af81d542e48fb45d43 (DEFAULT_TABLE table=test-table [id=ea574c8b265d481ea956c18eb6309acb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:35:46.472012 24690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51765401d74449af81d542e48fb45d43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:35:46.473590 24991 ts_tablet_manager.cc:1918] T a194f47e7fc340209896001c3bf902d4 P a6facfe4b14c490e8cc97f3d40b6d4df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:35:46.473757 24991 log.cc:1196] T a194f47e7fc340209896001c3bf902d4 P a6facfe4b14c490e8cc97f3d40b6d4df: Deleting WAL directory at /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/minicluster-data/ts-2/wal/wals/a194f47e7fc340209896001c3bf902d4
I20260620 18:35:46.474159 24991 ts_tablet_manager.cc:1939] T a194f47e7fc340209896001c3bf902d4 P a6facfe4b14c490e8cc97f3d40b6d4df: Deleting consensus metadata
I20260620 18:35:46.475716 24833 tablet_service.cc:1467] Processing CreateTablet for tablet 51765401d74449af81d542e48fb45d43 (DEFAULT_TABLE table=test-table [id=ea574c8b265d481ea956c18eb6309acb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:35:46.476110 24833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51765401d74449af81d542e48fb45d43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:35:46.478454 25001 tablet_bootstrap.cc:492] T 51765401d74449af81d542e48fb45d43 P 819af79e262244839ea1ff6ed5aec230: Bootstrap starting.
I20260620 18:35:46.478458 25000 tablet_bootstrap.cc:492] T 51765401d74449af81d542e48fb45d43 P f5ec1f95ec4544deb9e4ba19316d3e02: Bootstrap starting.
I20260620 18:35:46.479655 25000 tablet_bootstrap.cc:654] T 51765401d74449af81d542e48fb45d43 P f5ec1f95ec4544deb9e4ba19316d3e02: Neither blocks nor log segments found. Creating new log.
I20260620 18:35:46.480027 25001 tablet_bootstrap.cc:654] T 51765401d74449af81d542e48fb45d43 P 819af79e262244839ea1ff6ed5aec230: Neither blocks nor log segments found. Creating new log.
I20260620 18:35:46.481588 24399 catalog_manager.cc:4841] TS a6facfe4b14c490e8cc97f3d40b6d4df (127.22.37.3:35601): tablet a194f47e7fc340209896001c3bf902d4 (table test-table [id=99db8a67d95742499d5748a8d2ce933d]) successfully deleted
I20260620 18:35:46.481940 25000 tablet_bootstrap.cc:492] T 51765401d74449af81d542e48fb45d43 P f5ec1f95ec4544deb9e4ba19316d3e02: No bootstrap required, opened a new log
I20260620 18:35:46.482102 25000 ts_tablet_manager.cc:1397] T 51765401d74449af81d542e48fb45d43 P f5ec1f95ec4544deb9e4ba19316d3e02: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260620 18:35:46.482758 25002 tablet_bootstrap.cc:492] T 51765401d74449af81d542e48fb45d43 P a6facfe4b14c490e8cc97f3d40b6d4df: Bootstrap starting.
I20260620 18:35:46.482877 25000 raft_consensus.cc:348] T 51765401d74449af81d542e48fb45d43 P f5ec1f95ec4544deb9e4ba19316d3e02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ec1f95ec4544deb9e4ba19316d3e02" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 39519 } } peers { permanent_uuid: "819af79e262244839ea1ff6ed5aec230" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 33193 } } peers { permanent_uuid: "a6facfe4b14c490e8cc97f3d40b6d4df" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35601 } }
I20260620 18:35:46.483117 25000 raft_consensus.cc:374] T 51765401d74449af81d542e48fb45d43 P f5ec1f95ec4544deb9e4ba19316d3e02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:35:46.483201 25000 raft_consensus.cc:729] T 51765401d74449af81d542e48fb45d43 P f5ec1f95ec4544deb9e4ba19316d3e02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5ec1f95ec4544deb9e4ba19316d3e02, State: Initialized, Role: FOLLOWER
I20260620 18:35:46.483456 25000 consensus_queue.cc:260] T 51765401d74449af81d542e48fb45d43 P f5ec1f95ec4544deb9e4ba19316d3e02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ec1f95ec4544deb9e4ba19316d3e02" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 39519 } } peers { permanent_uuid: "819af79e262244839ea1ff6ed5aec230" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 33193 } } peers { permanent_uuid: "a6facfe4b14c490e8cc97f3d40b6d4df" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35601 } }
I20260620 18:35:46.484025 25001 tablet_bootstrap.cc:492] T 51765401d74449af81d542e48fb45d43 P 819af79e262244839ea1ff6ed5aec230: No bootstrap required, opened a new log
I20260620 18:35:46.484164 25001 ts_tablet_manager.cc:1397] T 51765401d74449af81d542e48fb45d43 P 819af79e262244839ea1ff6ed5aec230: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s
I20260620 18:35:46.484388 25000 ts_tablet_manager.cc:1428] T 51765401d74449af81d542e48fb45d43 P f5ec1f95ec4544deb9e4ba19316d3e02: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 18:35:46.484406 25002 tablet_bootstrap.cc:654] T 51765401d74449af81d542e48fb45d43 P a6facfe4b14c490e8cc97f3d40b6d4df: Neither blocks nor log segments found. Creating new log.
I20260620 18:35:46.485086 25001 raft_consensus.cc:348] T 51765401d74449af81d542e48fb45d43 P 819af79e262244839ea1ff6ed5aec230 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ec1f95ec4544deb9e4ba19316d3e02" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 39519 } } peers { permanent_uuid: "819af79e262244839ea1ff6ed5aec230" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 33193 } } peers { permanent_uuid: "a6facfe4b14c490e8cc97f3d40b6d4df" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35601 } }
I20260620 18:35:46.485638 25001 raft_consensus.cc:374] T 51765401d74449af81d542e48fb45d43 P 819af79e262244839ea1ff6ed5aec230 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:35:46.485788 25001 raft_consensus.cc:729] T 51765401d74449af81d542e48fb45d43 P 819af79e262244839ea1ff6ed5aec230 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819af79e262244839ea1ff6ed5aec230, State: Initialized, Role: FOLLOWER
I20260620 18:35:46.487798 25001 consensus_queue.cc:260] T 51765401d74449af81d542e48fb45d43 P 819af79e262244839ea1ff6ed5aec230 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ec1f95ec4544deb9e4ba19316d3e02" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 39519 } } peers { permanent_uuid: "819af79e262244839ea1ff6ed5aec230" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 33193 } } peers { permanent_uuid: "a6facfe4b14c490e8cc97f3d40b6d4df" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35601 } }
I20260620 18:35:46.488626 25001 ts_tablet_manager.cc:1428] T 51765401d74449af81d542e48fb45d43 P 819af79e262244839ea1ff6ed5aec230: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260620 18:35:46.493892 25002 tablet_bootstrap.cc:492] T 51765401d74449af81d542e48fb45d43 P a6facfe4b14c490e8cc97f3d40b6d4df: No bootstrap required, opened a new log
I20260620 18:35:46.494104 25002 ts_tablet_manager.cc:1397] T 51765401d74449af81d542e48fb45d43 P a6facfe4b14c490e8cc97f3d40b6d4df: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.007s
I20260620 18:35:46.494741 25002 raft_consensus.cc:348] T 51765401d74449af81d542e48fb45d43 P a6facfe4b14c490e8cc97f3d40b6d4df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ec1f95ec4544deb9e4ba19316d3e02" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 39519 } } peers { permanent_uuid: "819af79e262244839ea1ff6ed5aec230" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 33193 } } peers { permanent_uuid: "a6facfe4b14c490e8cc97f3d40b6d4df" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35601 } }
I20260620 18:35:46.494977 25002 raft_consensus.cc:374] T 51765401d74449af81d542e48fb45d43 P a6facfe4b14c490e8cc97f3d40b6d4df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:35:46.495059 25002 raft_consensus.cc:729] T 51765401d74449af81d542e48fb45d43 P a6facfe4b14c490e8cc97f3d40b6d4df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6facfe4b14c490e8cc97f3d40b6d4df, State: Initialized, Role: FOLLOWER
I20260620 18:35:46.495302 25002 consensus_queue.cc:260] T 51765401d74449af81d542e48fb45d43 P a6facfe4b14c490e8cc97f3d40b6d4df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ec1f95ec4544deb9e4ba19316d3e02" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 39519 } } peers { permanent_uuid: "819af79e262244839ea1ff6ed5aec230" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 33193 } } peers { permanent_uuid: "a6facfe4b14c490e8cc97f3d40b6d4df" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35601 } }
I20260620 18:35:46.495848 25002 ts_tablet_manager.cc:1428] T 51765401d74449af81d542e48fb45d43 P a6facfe4b14c490e8cc97f3d40b6d4df: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 18:35:46.682016 25007 raft_consensus.cc:482] T 51765401d74449af81d542e48fb45d43 P 819af79e262244839ea1ff6ed5aec230 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:35:46.682265 25007 raft_consensus.cc:504] T 51765401d74449af81d542e48fb45d43 P 819af79e262244839ea1ff6ed5aec230 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ec1f95ec4544deb9e4ba19316d3e02" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 39519 } } peers { permanent_uuid: "819af79e262244839ea1ff6ed5aec230" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 33193 } } peers { permanent_uuid: "a6facfe4b14c490e8cc97f3d40b6d4df" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35601 } }
I20260620 18:35:46.683851 25007 leader_election.cc:288] T 51765401d74449af81d542e48fb45d43 P 819af79e262244839ea1ff6ed5aec230 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f5ec1f95ec4544deb9e4ba19316d3e02 (127.22.37.1:39519), a6facfe4b14c490e8cc97f3d40b6d4df (127.22.37.3:35601)
I20260620 18:35:46.690701 25010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:46.684183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.2:41387 (local address 127.22.37.1:39519)
0620 18:35:46.684344 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:46.684348 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:46.684745 (+ 397us) server_negotiation.cc:407] Connection header received
0620 18:35:46.685065 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:46.685073 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:46.685177 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:46.685330 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:35:46.686070 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:46.686845 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:46.688934 (+ 2089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:46.689770 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:46.689913 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:46.690031 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:46.690289 (+ 258us) server_negotiation.cc:299] Negotiation successful
0620 18:35:46.690423 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 18:35:46.690879 25018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:46.684105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.1:39519 (local address 127.22.37.2:41387)
0620 18:35:46.684637 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:46.684658 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:46.684869 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:46.685589 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:46.685597 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:46.685627 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:35:46.685924 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:46.685933 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:46.686986 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:46.686993 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:46.688780 (+ 1787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:46.688792 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:46.689964 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:46.689970 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:46.690136 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:46.690158 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 18:35:46.690264 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 18:35:46.690430 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"mutex_wait_us":23,"thread_start_us":265,"threads_started":1}
I20260620 18:35:46.690961 25020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:46.684946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.2:45541 (local address 127.22.37.3:35601)
0620 18:35:46.685754 (+ 808us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:46.685760 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:46.685786 (+ 26us) server_negotiation.cc:407] Connection header received
0620 18:35:46.685859 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:46.685864 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:46.685928 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:46.686078 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:35:46.686591 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:46.687385 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:46.689101 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:46.689939 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:46.690084 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:46.690215 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:46.690488 (+ 273us) server_negotiation.cc:299] Negotiation successful
0620 18:35:46.690620 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":687,"thread_start_us":537,"threads_started":1}
I20260620 18:35:46.691009 25019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:46.684817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.3:35601 (local address 127.22.37.2:45541)
0620 18:35:46.685576 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:46.685617 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:46.685743 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:46.686043 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:46.686047 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:46.686071 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:35:46.686424 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:46.686433 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:46.687508 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:46.687514 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:46.688965 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:46.688976 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:46.690152 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:46.690159 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:46.690335 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:46.690358 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 18:35:46.690462 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 18:35:46.690587 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"mutex_wait_us":3,"thread_start_us":364,"threads_started":1}
I20260620 18:35:46.691708 24853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51765401d74449af81d542e48fb45d43" candidate_uuid: "819af79e262244839ea1ff6ed5aec230" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6facfe4b14c490e8cc97f3d40b6d4df" is_pre_election: true
I20260620 18:35:46.691957 24853 raft_consensus.cc:2454] T 51765401d74449af81d542e48fb45d43 P a6facfe4b14c490e8cc97f3d40b6d4df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 819af79e262244839ea1ff6ed5aec230 in term 0.
I20260620 18:35:46.691843 24576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51765401d74449af81d542e48fb45d43" candidate_uuid: "819af79e262244839ea1ff6ed5aec230" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5ec1f95ec4544deb9e4ba19316d3e02" is_pre_election: true
I20260620 18:35:46.692261 24576 raft_consensus.cc:2454] T 51765401d74449af81d542e48fb45d43 P f5ec1f95ec4544deb9e4ba19316d3e02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 819af79e262244839ea1ff6ed5aec230 in term 0.
I20260620 18:35:46.692466 24642 leader_election.cc:302] T 51765401d74449af81d542e48fb45d43 P 819af79e262244839ea1ff6ed5aec230 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 819af79e262244839ea1ff6ed5aec230, a6facfe4b14c490e8cc97f3d40b6d4df; no voters:
I20260620 18:35:46.692939 25007 raft_consensus.cc:2781] T 51765401d74449af81d542e48fb45d43 P 819af79e262244839ea1ff6ed5aec230 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:35:46.693060 25007 raft_consensus.cc:482] T 51765401d74449af81d542e48fb45d43 P 819af79e262244839ea1ff6ed5aec230 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:35:46.693207 25007 raft_consensus.cc:3037] T 51765401d74449af81d542e48fb45d43 P 819af79e262244839ea1ff6ed5aec230 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:35:46.695398 25007 raft_consensus.cc:504] T 51765401d74449af81d542e48fb45d43 P 819af79e262244839ea1ff6ed5aec230 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ec1f95ec4544deb9e4ba19316d3e02" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 39519 } } peers { permanent_uuid: "819af79e262244839ea1ff6ed5aec230" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 33193 } } peers { permanent_uuid: "a6facfe4b14c490e8cc97f3d40b6d4df" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35601 } }
I20260620 18:35:46.696007 25007 leader_election.cc:288] T 51765401d74449af81d542e48fb45d43 P 819af79e262244839ea1ff6ed5aec230 [CANDIDATE]: Term 1 election: Requested vote from peers f5ec1f95ec4544deb9e4ba19316d3e02 (127.22.37.1:39519), a6facfe4b14c490e8cc97f3d40b6d4df (127.22.37.3:35601)
I20260620 18:35:46.696518 24576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51765401d74449af81d542e48fb45d43" candidate_uuid: "819af79e262244839ea1ff6ed5aec230" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5ec1f95ec4544deb9e4ba19316d3e02"
I20260620 18:35:46.696681 24576 raft_consensus.cc:3037] T 51765401d74449af81d542e48fb45d43 P f5ec1f95ec4544deb9e4ba19316d3e02 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:35:46.696616 24853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51765401d74449af81d542e48fb45d43" candidate_uuid: "819af79e262244839ea1ff6ed5aec230" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6facfe4b14c490e8cc97f3d40b6d4df"
I20260620 18:35:46.696877 24853 raft_consensus.cc:3037] T 51765401d74449af81d542e48fb45d43 P a6facfe4b14c490e8cc97f3d40b6d4df [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:35:46.699371 24853 raft_consensus.cc:2454] T 51765401d74449af81d542e48fb45d43 P a6facfe4b14c490e8cc97f3d40b6d4df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 819af79e262244839ea1ff6ed5aec230 in term 1.
I20260620 18:35:46.699898 24576 raft_consensus.cc:2454] T 51765401d74449af81d542e48fb45d43 P f5ec1f95ec4544deb9e4ba19316d3e02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 819af79e262244839ea1ff6ed5aec230 in term 1.
I20260620 18:35:46.699944 24642 leader_election.cc:302] T 51765401d74449af81d542e48fb45d43 P 819af79e262244839ea1ff6ed5aec230 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 819af79e262244839ea1ff6ed5aec230, a6facfe4b14c490e8cc97f3d40b6d4df; no voters:
I20260620 18:35:46.700214 25007 raft_consensus.cc:2781] T 51765401d74449af81d542e48fb45d43 P 819af79e262244839ea1ff6ed5aec230 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:35:46.700735 25007 raft_consensus.cc:686] T 51765401d74449af81d542e48fb45d43 P 819af79e262244839ea1ff6ed5aec230 [term 1 LEADER]: Becoming Leader. State: Replica: 819af79e262244839ea1ff6ed5aec230, State: Running, Role: LEADER
I20260620 18:35:46.701223 25007 consensus_queue.cc:237] T 51765401d74449af81d542e48fb45d43 P 819af79e262244839ea1ff6ed5aec230 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ec1f95ec4544deb9e4ba19316d3e02" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 39519 } } peers { permanent_uuid: "819af79e262244839ea1ff6ed5aec230" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 33193 } } peers { permanent_uuid: "a6facfe4b14c490e8cc97f3d40b6d4df" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35601 } }
I20260620 18:35:46.704574 24413 catalog_manager.cc:5495] T 51765401d74449af81d542e48fb45d43 P 819af79e262244839ea1ff6ed5aec230 reported cstate change: term changed from 0 to 1, leader changed from <none> to 819af79e262244839ea1ff6ed5aec230 (127.22.37.2). New cstate: current_term: 1 leader_uuid: "819af79e262244839ea1ff6ed5aec230" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ec1f95ec4544deb9e4ba19316d3e02" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 39519 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "819af79e262244839ea1ff6ed5aec230" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 33193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6facfe4b14c490e8cc97f3d40b6d4df" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35601 } health_report { overall_health: UNKNOWN } } }
I20260620 18:35:46.728873 25023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:46.722987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57092 (local address 127.22.37.2:33193)
0620 18:35:46.723578 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:46.723585 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:46.723603 (+ 18us) server_negotiation.cc:407] Connection header received
0620 18:35:46.723689 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:46.723692 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:46.723757 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:46.723932 (+ 175us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:35:46.724331 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:46.725279 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:46.726575 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:46.727005 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:46.727135 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:46.727639 (+ 504us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:35:46.728229 (+ 590us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:35:46.728395 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:46.728443 (+ 48us) server_negotiation.cc:299] Negotiation successful
0620 18:35:46.728515 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":309,"threads_started":1}
I20260620 18:35:46.743880 25026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:46.738818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.1:39519 (local address 127.22.37.2:41203)
0620 18:35:46.739373 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:46.739389 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:46.739516 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:46.739796 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:46.739799 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:46.739807 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 18:35:46.740066 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:46.740072 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:46.740999 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:46.741003 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:46.742107 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:46.742118 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:46.742756 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:46.742760 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:46.742869 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:46.742914 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 18:35:46.743256 (+ 342us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 18:35:46.743268 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 18:35:46.743349 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 18:35:46.743476 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":296,"threads_started":1}
I20260620 18:35:46.744037 25010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:46.738864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.2:41203 (local address 127.22.37.1:39519)
0620 18:35:46.739098 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:46.739108 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:46.739473 (+ 365us) server_negotiation.cc:407] Connection header received
0620 18:35:46.739633 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:46.739637 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:46.739701 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:46.739772 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:35:46.740183 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:46.740880 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:46.742227 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:46.742619 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:46.742709 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:46.743046 (+ 337us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:35:46.743133 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:35:46.743194 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:46.743525 (+ 331us) server_negotiation.cc:299] Negotiation successful
0620 18:35:46.743633 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 18:35:46.752727 24853 raft_consensus.cc:1261] T 51765401d74449af81d542e48fb45d43 P a6facfe4b14c490e8cc97f3d40b6d4df [term 1 FOLLOWER]: Refusing update from remote peer 819af79e262244839ea1ff6ed5aec230: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:35:46.753083 24576 raft_consensus.cc:1261] T 51765401d74449af81d542e48fb45d43 P f5ec1f95ec4544deb9e4ba19316d3e02 [term 1 FOLLOWER]: Refusing update from remote peer 819af79e262244839ea1ff6ed5aec230: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:35:46.753458 25021 consensus_queue.cc:1035] T 51765401d74449af81d542e48fb45d43 P 819af79e262244839ea1ff6ed5aec230 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6facfe4b14c490e8cc97f3d40b6d4df" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:35:46.753836 25007 consensus_queue.cc:1035] T 51765401d74449af81d542e48fb45d43 P 819af79e262244839ea1ff6ed5aec230 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5ec1f95ec4544deb9e4ba19316d3e02" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 39519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:35:46.780642 25023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:46.774636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.1:40181 (local address 127.22.37.2:33193)
0620 18:35:46.774783 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:46.774787 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:46.775328 (+ 541us) server_negotiation.cc:407] Connection header received
0620 18:35:46.775447 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:46.775450 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:46.775507 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:46.775602 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:35:46.776211 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:46.777160 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:46.778983 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:46.779392 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:46.779492 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:46.779857 (+ 365us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:35:46.780023 (+ 166us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:35:46.780114 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:46.780250 (+ 136us) server_negotiation.cc:299] Negotiation successful
0620 18:35:46.780325 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 18:35:46.780776 25031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:46.774621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.2:33193 (local address 127.22.37.1:40181)
0620 18:35:46.775232 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:46.775262 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:46.775367 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:46.775646 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:46.775650 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:46.775663 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 18:35:46.776002 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:46.776014 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:46.777297 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:46.777300 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:46.778842 (+ 1542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:46.778853 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:46.779556 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:46.779561 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:46.779672 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:46.779708 (+ 36us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 18:35:46.780123 (+ 415us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 18:35:46.780133 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 18:35:46.780225 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 18:35:46.780349 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":344,"threads_started":1}
W20260620 18:35:46.781975 24973 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 18:35:48.373734 24412 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39066:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:35:48.373940 24412 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39066:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:35:48.377743 24412 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 518479382dc0411d87ae9ad184d6250a: Sending DeleteTablet for 3 replicas of tablet 51765401d74449af81d542e48fb45d43
I20260620 18:35:48.378540 24690 tablet_service.cc:1514] Processing DeleteTablet for tablet 51765401d74449af81d542e48fb45d43 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:35:48 UTC) from {username='kudu', principal='kudu/127.22.37.62@KRBTEST.COM'} at 127.0.0.1:57084
I20260620 18:35:48.378530 24551 tablet_service.cc:1514] Processing DeleteTablet for tablet 51765401d74449af81d542e48fb45d43 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:35:48 UTC) from {username='kudu', principal='kudu/127.22.37.62@KRBTEST.COM'} at 127.0.0.1:54838
I20260620 18:35:48.378609 24833 tablet_service.cc:1514] Processing DeleteTablet for tablet 51765401d74449af81d542e48fb45d43 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:35:48 UTC) from {username='kudu', principal='kudu/127.22.37.62@KRBTEST.COM'} at 127.0.0.1:60390
I20260620 18:35:48.379344 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 24485
I20260620 18:35:48.391498 25056 tablet_replica.cc:330] T 51765401d74449af81d542e48fb45d43 P a6facfe4b14c490e8cc97f3d40b6d4df: stopping tablet replica
I20260620 18:35:48.392136 25054 tablet_replica.cc:330] T 51765401d74449af81d542e48fb45d43 P 819af79e262244839ea1ff6ed5aec230: stopping tablet replica
I20260620 18:35:48.392483 25056 raft_consensus.cc:2229] T 51765401d74449af81d542e48fb45d43 P a6facfe4b14c490e8cc97f3d40b6d4df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:35:48.392482 25054 raft_consensus.cc:2229] T 51765401d74449af81d542e48fb45d43 P 819af79e262244839ea1ff6ed5aec230 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:35:48.392794 25056 raft_consensus.cc:2258] T 51765401d74449af81d542e48fb45d43 P a6facfe4b14c490e8cc97f3d40b6d4df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:35:48.393172 25055 tablet_replica.cc:330] T 51765401d74449af81d542e48fb45d43 P f5ec1f95ec4544deb9e4ba19316d3e02: stopping tablet replica
I20260620 18:35:48.393487 25055 raft_consensus.cc:2229] T 51765401d74449af81d542e48fb45d43 P f5ec1f95ec4544deb9e4ba19316d3e02 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:35:48.393755 25055 raft_consensus.cc:2258] T 51765401d74449af81d542e48fb45d43 P f5ec1f95ec4544deb9e4ba19316d3e02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:35:48.393769 25054 raft_consensus.cc:2258] T 51765401d74449af81d542e48fb45d43 P 819af79e262244839ea1ff6ed5aec230 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:35:48.394742 25055 ts_tablet_manager.cc:1905] T 51765401d74449af81d542e48fb45d43 P f5ec1f95ec4544deb9e4ba19316d3e02: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:35:48.394739 25056 ts_tablet_manager.cc:1905] T 51765401d74449af81d542e48fb45d43 P a6facfe4b14c490e8cc97f3d40b6d4df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:35:48.395287 25054 ts_tablet_manager.cc:1905] T 51765401d74449af81d542e48fb45d43 P 819af79e262244839ea1ff6ed5aec230: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:35:48.401453 25056 ts_tablet_manager.cc:1918] T 51765401d74449af81d542e48fb45d43 P a6facfe4b14c490e8cc97f3d40b6d4df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 18:35:48.401644 25056 log.cc:1196] T 51765401d74449af81d542e48fb45d43 P a6facfe4b14c490e8cc97f3d40b6d4df: Deleting WAL directory at /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/minicluster-data/ts-2/wal/wals/51765401d74449af81d542e48fb45d43
I20260620 18:35:48.401925 25055 ts_tablet_manager.cc:1918] T 51765401d74449af81d542e48fb45d43 P f5ec1f95ec4544deb9e4ba19316d3e02: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 18:35:48.401957 25054 ts_tablet_manager.cc:1918] T 51765401d74449af81d542e48fb45d43 P 819af79e262244839ea1ff6ed5aec230: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 18:35:48.402067 25054 log.cc:1196] T 51765401d74449af81d542e48fb45d43 P 819af79e262244839ea1ff6ed5aec230: Deleting WAL directory at /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/minicluster-data/ts-1/wal/wals/51765401d74449af81d542e48fb45d43
I20260620 18:35:48.402215 25056 ts_tablet_manager.cc:1939] T 51765401d74449af81d542e48fb45d43 P a6facfe4b14c490e8cc97f3d40b6d4df: Deleting consensus metadata
I20260620 18:35:48.402235 25055 log.cc:1196] T 51765401d74449af81d542e48fb45d43 P f5ec1f95ec4544deb9e4ba19316d3e02: Deleting WAL directory at /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980527431848-22676-0/minicluster-data/ts-0/wal/wals/51765401d74449af81d542e48fb45d43
I20260620 18:35:48.402513 25054 ts_tablet_manager.cc:1939] T 51765401d74449af81d542e48fb45d43 P 819af79e262244839ea1ff6ed5aec230: Deleting consensus metadata
Jun 20 18:35:48 dist-test-slave-5lgp krb5kdc[24369](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980546, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.22.37.1@KRBTEST.COM
I20260620 18:35:48.402846 25055 ts_tablet_manager.cc:1939] T 51765401d74449af81d542e48fb45d43 P f5ec1f95ec4544deb9e4ba19316d3e02: Deleting consensus metadata
I20260620 18:35:48.403354 24399 catalog_manager.cc:4841] TS a6facfe4b14c490e8cc97f3d40b6d4df (127.22.37.3:35601): tablet 51765401d74449af81d542e48fb45d43 (table test-table [id=ea574c8b265d481ea956c18eb6309acb]) successfully deleted
I20260620 18:35:48.403882 24399 catalog_manager.cc:4841] TS f5ec1f95ec4544deb9e4ba19316d3e02 (127.22.37.1:39519): tablet 51765401d74449af81d542e48fb45d43 (table test-table [id=ea574c8b265d481ea956c18eb6309acb]) successfully deleted
I20260620 18:35:48.403941 24398 catalog_manager.cc:4841] TS 819af79e262244839ea1ff6ed5aec230 (127.22.37.2:33193): tablet 51765401d74449af81d542e48fb45d43 (table test-table [id=ea574c8b265d481ea956c18eb6309acb]) successfully deleted
I20260620 18:35:48.409763 25058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:48.382146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50080 (local address 127.22.37.1:39519)
0620 18:35:48.387290 (+ 5144us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:48.387299 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:48.392531 (+ 5232us) server_negotiation.cc:407] Connection header received
0620 18:35:48.392776 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:48.392789 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:48.392885 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:48.393019 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:48.394777 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:48.395918 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:48.397783 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:48.398368 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:48.398495 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:48.404180 (+ 5685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:48.404194 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:48.404211 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:48.404240 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:48.407497 (+ 3257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:48.408064 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:48.408071 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:48.408074 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:48.408186 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:48.408496 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:48.408499 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:48.408501 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:48.408831 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:48.409028 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:48.409261 (+ 233us) server_negotiation.cc:299] Negotiation successful
0620 18:35:48.409502 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4964,"thread_start_us":4797,"threads_started":1}
W20260620 18:35:48.410408 24616 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50080
W20260620 18:35:48.411005 22676 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 24485: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:35:48.411233 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 24485
I20260620 18:35:48.436681 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 24625
Jun 20 18:35:48 dist-test-slave-5lgp krb5kdc[24369](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980546, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.22.37.2@KRBTEST.COM
I20260620 18:35:48.453096 25059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:48.437512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42140 (local address 127.22.37.2:33193)
0620 18:35:48.438096 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:48.438100 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:48.438116 (+ 16us) server_negotiation.cc:407] Connection header received
0620 18:35:48.438174 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:48.438178 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:48.438242 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:48.438372 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:48.440242 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:48.441032 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:48.442511 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:48.442963 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:48.443105 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:48.447457 (+ 4352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:48.447477 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:48.447494 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:48.447528 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:48.450362 (+ 2834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:48.451015 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:48.451022 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:48.451025 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:48.451148 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:48.451548 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:48.451551 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:48.451553 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:48.451872 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:48.452195 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:48.452467 (+ 272us) server_negotiation.cc:299] Negotiation successful
0620 18:35:48.452699 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":308,"threads_started":1}
W20260620 18:35:48.453843 24755 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42140
W20260620 18:35:48.454522 22676 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 24625: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:35:48.454700 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 24625
I20260620 18:35:48.476619 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 24764
Jun 20 18:35:48 dist-test-slave-5lgp krb5kdc[24369](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980546, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.22.37.3@KRBTEST.COM
I20260620 18:35:48.493151 25060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:48.477444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36258 (local address 127.22.37.3:35601)
0620 18:35:48.478002 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:48.478006 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:48.478022 (+ 16us) server_negotiation.cc:407] Connection header received
0620 18:35:48.478078 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:48.478086 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:48.478149 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:48.478244 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:48.479514 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:48.480433 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:48.481725 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:48.482121 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:48.482220 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:48.486672 (+ 4452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:48.486686 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:48.486704 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:48.486739 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:48.490126 (+ 3387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:48.490883 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:48.490891 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:48.490894 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:48.491041 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:48.491459 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:48.491469 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:48.491471 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:48.491851 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:48.492158 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:48.492543 (+ 385us) server_negotiation.cc:299] Negotiation successful
0620 18:35:48.492775 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":299,"threads_started":1}
W20260620 18:35:48.493875 24894 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36258
W20260620 18:35:48.494526 22676 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 24764: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:35:48.494729 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 24764
I20260620 18:35:48.511950 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 24382
I20260620 18:35:48.513235 24454 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:35:48.627583 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 24382
2026-06-20T18:35:48Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8484 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980527431848-22676-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:35:48 dist-test-slave-5lgp krb5kdc[25064](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:35:48 dist-test-slave-5lgp krb5kdc[25064](info): set up 2 sockets
Jun 20 18:35:48 dist-test-slave-5lgp krb5kdc[25064](info): commencing operation
krb5kdc: starting...
W20260620 18:35:49.709676 22676 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980527431848-22676-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980527431848-22676-0/krb5kdc/test-user.keytab.
Jun 20 18:35:49 dist-test-slave-5lgp krb5kdc[25064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980549, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:35:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:35:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.37.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.37.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.37.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab.
Entry for principal kudu/127.22.37.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab.
I20260620 18:35:49.895087 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980527431848-22676-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980527431848-22676-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980527431848-22676-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.62:35633
--webserver_interface=127.22.37.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab
--principal=kudu/127.22.37.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:43723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.37.62:35633
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:35:50.222839 25077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:35:50.223125 25077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:35:50.223284 25077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:35:50.230461 25077 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:35:50.230576 25077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:35:50.230648 25077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:35:50.230733 25077 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:35:50.230815 25077 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:35:50.240645 25077 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:43723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980527431848-22676-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980527431848-22676-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.37.62:35633
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab
--principal=kudu/127.22.37.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.62:35633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.37.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980527431848-22676-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:35:50.242027 25077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:35:50.243361 25077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:35:50.255200 25082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:50.255198 25085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:50.255378 25083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:35:50.255905 25077 server_base.cc:1034] running on GCE node
I20260620 18:35:50.256634 25077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:35:50.258057 25077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:35:50.259349 25077 hybrid_clock.cc:648] HybridClock initialized: now 1781980550259288 us; error 60 us; skew 500 ppm
I20260620 18:35:50.259697 25077 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:35:50 dist-test-slave-5lgp krb5kdc[25064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980550, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:35:50.262660 25077 init.cc:377] Logged in from keytab as kudu/127.22.37.62@KRBTEST.COM (short username kudu)
I20260620 18:35:50.264148 25077 webserver.cc:460] Webserver started at http://127.22.37.62:32777/ using document root <none> and password file <none>
I20260620 18:35:50.264823 25077 fs_manager.cc:359] Metadata directory not provided
I20260620 18:35:50.264976 25077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:35:50.265293 25077 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:35:50.268181 25077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980527431848-22676-0/minicluster-data/master-0/data/instance:
uuid: "ccaab220eb214fe9995f5ac32d80dd2d"
format_stamp: "Formatted at 2026-06-20 18:35:50 on dist-test-slave-5lgp"
server_key: "b8a80729e6cb60b164c0bed28148e808"
server_key_iv: "e441340c9042810d503bbd6ab9af4e23"
server_key_version: "encryptionkey@0"
I20260620 18:35:50.268904 25077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980527431848-22676-0/minicluster-data/master-0/wal/instance:
uuid: "ccaab220eb214fe9995f5ac32d80dd2d"
format_stamp: "Formatted at 2026-06-20 18:35:50 on dist-test-slave-5lgp"
server_key: "b8a80729e6cb60b164c0bed28148e808"
server_key_iv: "e441340c9042810d503bbd6ab9af4e23"
server_key_version: "encryptionkey@0"
I20260620 18:35:50.273857 25077 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260620 18:35:50.277010 25092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:50.278406 25077 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:35:50.278581 25077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980527431848-22676-0/minicluster-data/master-0/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980527431848-22676-0/minicluster-data/master-0/wal
uuid: "ccaab220eb214fe9995f5ac32d80dd2d"
format_stamp: "Formatted at 2026-06-20 18:35:50 on dist-test-slave-5lgp"
server_key: "b8a80729e6cb60b164c0bed28148e808"
server_key_iv: "e441340c9042810d503bbd6ab9af4e23"
server_key_version: "encryptionkey@0"
I20260620 18:35:50.278774 25077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980527431848-22676-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980527431848-22676-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980527431848-22676-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:35:50.294689 25077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:35:50.298630 25077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:35:50.298907 25077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:35:50.316725 25077 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.62:35633
I20260620 18:35:50.316761 25143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.62:35633 every 8 connection(s)
I20260620 18:35:50.318094 25077 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
I20260620 18:35:50.321715 25144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:35:50.322697 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 25077
I20260620 18:35:50.322943 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980527431848-22676-0/minicluster-data/master-0/wal/instance
I20260620 18:35:50.323556 22676 external_mini_cluster.cc:1428] Setting key 92822d03cce14a9b4eea94f8ab62c222
I20260620 18:35:50.335620 25144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccaab220eb214fe9995f5ac32d80dd2d: Bootstrap starting.
Jun 20 18:35:50 dist-test-slave-5lgp krb5kdc[25064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980549, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM
I20260620 18:35:50.338578 25144 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ccaab220eb214fe9995f5ac32d80dd2d: Neither blocks nor log segments found. Creating new log.
I20260620 18:35:50.339906 25144 log.cc:824] T 00000000000000000000000000000000 P ccaab220eb214fe9995f5ac32d80dd2d: Log is configured to *not* fsync() on all Append() calls
I20260620 18:35:50.343143 25144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccaab220eb214fe9995f5ac32d80dd2d: No bootstrap required, opened a new log
I20260620 18:35:50.347098 25147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:50.325536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36056 (local address 127.22.37.62:35633)
0620 18:35:50.326392 (+ 856us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:50.326407 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:50.326459 (+ 52us) server_negotiation.cc:407] Connection header received
0620 18:35:50.328161 (+ 1702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:50.328190 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:50.328740 (+ 550us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:50.329188 (+ 448us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:50.330561 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:50.332550 (+ 1989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:50.333914 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:50.334914 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:50.335079 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:50.338777 (+ 3698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:50.338788 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:50.338805 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:50.338848 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:50.342709 (+ 3861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:50.343340 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:50.343348 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:50.343356 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:50.343571 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:50.343937 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:50.343947 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:50.343952 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:50.344626 (+ 674us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:50.344904 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:50.345277 (+ 373us) server_negotiation.cc:299] Negotiation successful
0620 18:35:50.345751 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":340,"threads_started":1}
I20260620 18:35:50.352391 25144 raft_consensus.cc:348] T 00000000000000000000000000000000 P ccaab220eb214fe9995f5ac32d80dd2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccaab220eb214fe9995f5ac32d80dd2d" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 35633 } }
I20260620 18:35:50.352715 25144 raft_consensus.cc:374] T 00000000000000000000000000000000 P ccaab220eb214fe9995f5ac32d80dd2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:35:50.352794 25144 raft_consensus.cc:729] T 00000000000000000000000000000000 P ccaab220eb214fe9995f5ac32d80dd2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccaab220eb214fe9995f5ac32d80dd2d, State: Initialized, Role: FOLLOWER
I20260620 18:35:50.353317 25144 consensus_queue.cc:260] T 00000000000000000000000000000000 P ccaab220eb214fe9995f5ac32d80dd2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccaab220eb214fe9995f5ac32d80dd2d" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 35633 } }
I20260620 18:35:50.353605 25144 raft_consensus.cc:388] T 00000000000000000000000000000000 P ccaab220eb214fe9995f5ac32d80dd2d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:35:50.353746 25144 raft_consensus.cc:482] T 00000000000000000000000000000000 P ccaab220eb214fe9995f5ac32d80dd2d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:35:50.353927 25144 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ccaab220eb214fe9995f5ac32d80dd2d [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:35:50.356154 25144 raft_consensus.cc:504] T 00000000000000000000000000000000 P ccaab220eb214fe9995f5ac32d80dd2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccaab220eb214fe9995f5ac32d80dd2d" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 35633 } }
I20260620 18:35:50.356658 25144 leader_election.cc:302] T 00000000000000000000000000000000 P ccaab220eb214fe9995f5ac32d80dd2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ccaab220eb214fe9995f5ac32d80dd2d; no voters:
I20260620 18:35:50.357295 25144 leader_election.cc:288] T 00000000000000000000000000000000 P ccaab220eb214fe9995f5ac32d80dd2d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:35:50.357466 25149 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ccaab220eb214fe9995f5ac32d80dd2d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:35:50.358130 25149 raft_consensus.cc:686] T 00000000000000000000000000000000 P ccaab220eb214fe9995f5ac32d80dd2d [term 1 LEADER]: Becoming Leader. State: Replica: ccaab220eb214fe9995f5ac32d80dd2d, State: Running, Role: LEADER
I20260620 18:35:50.358650 25149 consensus_queue.cc:237] T 00000000000000000000000000000000 P ccaab220eb214fe9995f5ac32d80dd2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccaab220eb214fe9995f5ac32d80dd2d" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 35633 } }
I20260620 18:35:50.359390 25144 sys_catalog.cc:564] T 00000000000000000000000000000000 P ccaab220eb214fe9995f5ac32d80dd2d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:35:50.361358 25150 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccaab220eb214fe9995f5ac32d80dd2d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ccaab220eb214fe9995f5ac32d80dd2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccaab220eb214fe9995f5ac32d80dd2d" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 35633 } } }
I20260620 18:35:50.361850 25150 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccaab220eb214fe9995f5ac32d80dd2d [sys.catalog]: This master's current role is: LEADER
I20260620 18:35:50.362485 25151 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccaab220eb214fe9995f5ac32d80dd2d [sys.catalog]: SysCatalogTable state changed. Reason: New leader ccaab220eb214fe9995f5ac32d80dd2d. Latest consensus state: current_term: 1 leader_uuid: "ccaab220eb214fe9995f5ac32d80dd2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccaab220eb214fe9995f5ac32d80dd2d" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 35633 } } }
I20260620 18:35:50.362749 25151 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccaab220eb214fe9995f5ac32d80dd2d [sys.catalog]: This master's current role is: LEADER
I20260620 18:35:50.364261 25156 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:35:50.369009 25156 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:35:50.377246 25156 catalog_manager.cc:1347] Generated new cluster ID: 7d33324dbc904ed9a238d3c8000497dc
I20260620 18:35:50.377410 25156 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:35:50.388374 25156 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:35:50.389755 25156 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:35:50.400225 25156 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ccaab220eb214fe9995f5ac32d80dd2d: Generated new TSK 0
I20260620 18:35:50.401048 25156 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.37.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.37.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.37.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.1.keytab.
Entry for principal kudu/127.22.37.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.1.keytab.
I20260620 18:35:50.449057 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980527431848-22676-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980527431848-22676-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980527431848-22676-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980527431848-22676-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.37.1:0
--local_ip_for_outbound_sockets=127.22.37.1
--webserver_interface=127.22.37.1
--webserver_port=0
--tserver_master_addrs=127.22.37.62:35633
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.1.keytab
--principal=kudu/127.22.37.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:43723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:35:50.771310 25170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:35:50.771582 25170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:35:50.771685 25170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:35:50.778822 25170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:35:50.778935 25170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:35:50.779049 25170 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.37.1
I20260620 18:35:50.789114 25170 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:43723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980527431848-22676-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980527431848-22676-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.1.keytab
--principal=kudu/127.22.37.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.37.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980527431848-22676-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.37.1
--webserver_port=0
--tserver_master_addrs=127.22.37.62:35633
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.37.1
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980527431848-22676-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:35:50.790482 25170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:35:50.791695 25170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:35:50.804826 25176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:50.804865 25178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:50.804921 25175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:35:50.805624 25170 server_base.cc:1034] running on GCE node
I20260620 18:35:50.806253 25170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:35:50.807291 25170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:35:50.808518 25170 hybrid_clock.cc:648] HybridClock initialized: now 1781980550808468 us; error 49 us; skew 500 ppm
I20260620 18:35:50.808835 25170 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:35:50 dist-test-slave-5lgp krb5kdc[25064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980550, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:35:50.812021 25170 init.cc:377] Logged in from keytab as kudu/127.22.37.1@KRBTEST.COM (short username kudu)
I20260620 18:35:50.813515 25170 webserver.cc:460] Webserver started at http://127.22.37.1:36131/ using document root <none> and password file <none>
I20260620 18:35:50.814148 25170 fs_manager.cc:359] Metadata directory not provided
I20260620 18:35:50.814241 25170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:35:50.814486 25170 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:35:50.817294 25170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980527431848-22676-0/minicluster-data/ts-0/data/instance:
uuid: "cde685fa6ef54e54ad3f077977466219"
format_stamp: "Formatted at 2026-06-20 18:35:50 on dist-test-slave-5lgp"
server_key: "c26c91a74224eb3174a1e0ecb556250f"
server_key_iv: "334e8c87934620ad91dc571d04c72aa3"
server_key_version: "encryptionkey@0"
I20260620 18:35:50.818080 25170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980527431848-22676-0/minicluster-data/ts-0/wal/instance:
uuid: "cde685fa6ef54e54ad3f077977466219"
format_stamp: "Formatted at 2026-06-20 18:35:50 on dist-test-slave-5lgp"
server_key: "c26c91a74224eb3174a1e0ecb556250f"
server_key_iv: "334e8c87934620ad91dc571d04c72aa3"
server_key_version: "encryptionkey@0"
I20260620 18:35:50.822973 25170 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 18:35:50.826164 25185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:50.827598 25170 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:35:50.827730 25170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980527431848-22676-0/minicluster-data/ts-0/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980527431848-22676-0/minicluster-data/ts-0/wal
uuid: "cde685fa6ef54e54ad3f077977466219"
format_stamp: "Formatted at 2026-06-20 18:35:50 on dist-test-slave-5lgp"
server_key: "c26c91a74224eb3174a1e0ecb556250f"
server_key_iv: "334e8c87934620ad91dc571d04c72aa3"
server_key_version: "encryptionkey@0"
I20260620 18:35:50.827903 25170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980527431848-22676-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980527431848-22676-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980527431848-22676-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:35:50.850940 25170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:35:50.855309 25170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:35:50.855544 25170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:35:50.856485 25170 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:35:50.857760 25170 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:35:50.857831 25170 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:50.857933 25170 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:35:50.857978 25170 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:50.890326 25170 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.1:40971
I20260620 18:35:50.890420 25297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.1:40971 every 8 connection(s)
I20260620 18:35:50.891633 25170 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980527431848-22676-0/minicluster-data/ts-0/data/info.pb
I20260620 18:35:50.897003 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 25170
I20260620 18:35:50.897207 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980527431848-22676-0/minicluster-data/ts-0/wal/instance
I20260620 18:35:50.898041 22676 external_mini_cluster.cc:1428] Setting key e846bb8d680ec11b5e8bcac69f7c0f25
Jun 20 18:35:50 dist-test-slave-5lgp krb5kdc[25064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980550, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.1@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM
I20260620 18:35:50.913095 25300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:50.894159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.1:47639 (local address 127.22.37.62:35633)
0620 18:35:50.894576 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:50.894583 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:50.895626 (+ 1043us) server_negotiation.cc:407] Connection header received
0620 18:35:50.897877 (+ 2251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:50.897882 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:50.897946 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:50.898042 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:50.899941 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:50.900793 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:50.902350 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:50.902730 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:50.902857 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:50.907014 (+ 4157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:50.907019 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:50.907022 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:50.907057 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:50.909468 (+ 2411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:50.910380 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:50.910385 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:50.910387 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:50.910534 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:50.911025 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:50.911038 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:50.911040 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:50.911347 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:50.911466 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:50.912363 (+ 897us) server_negotiation.cc:299] Negotiation successful
0620 18:35:50.912606 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":189,"threads_started":1}
I20260620 18:35:50.913769 25301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:50.894788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:35633 (local address 127.22.37.1:47639)
0620 18:35:50.895466 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:50.895508 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:50.897568 (+ 2060us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:50.898315 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:50.898327 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:50.899018 (+ 691us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:50.899785 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:50.899804 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:50.900936 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:50.900941 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:50.902208 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:50.902219 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:50.902944 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:50.902951 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:50.903154 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:50.903825 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:35:50.903854 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:50.906810 (+ 2956us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:35:50.909590 (+ 2780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:50.909602 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:50.909620 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:50.910221 (+ 601us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:50.910656 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:50.910664 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:50.910671 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:50.910905 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:50.911485 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:35:50.911496 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:35:50.911927 (+ 431us) client_negotiation.cc:769] Sending connection context
0620 18:35:50.912235 (+ 308us) client_negotiation.cc:240] Negotiation successful
0620 18:35:50.912672 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":266,"threads_started":1}
I20260620 18:35:50.915179 25298 heartbeater.cc:347] Connected to a master server at 127.22.37.62:35633
I20260620 18:35:50.915522 25298 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.22.37.2@KRBTEST.COM; defaulting to no policy
I20260620 18:35:50.916379 25298 heartbeater.cc:511] Master 127.22.37.62:35633 requested a full tablet report, sending...
I20260620 18:35:50.918495 25109 ts_manager.cc:194] Registered new tserver with Master: cde685fa6ef54e54ad3f077977466219 (127.22.37.1:40971)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.37.2@KRBTEST.COM" created.
I20260620 18:35:50.919927 25109 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.37.1@KRBTEST.COM'} at 127.22.37.1:47639
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.37.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.2.keytab.
Entry for principal kudu/127.22.37.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.2.keytab.
I20260620 18:35:50.940891 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980527431848-22676-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980527431848-22676-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980527431848-22676-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980527431848-22676-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.37.2:0
--local_ip_for_outbound_sockets=127.22.37.2
--webserver_interface=127.22.37.2
--webserver_port=0
--tserver_master_addrs=127.22.37.62:35633
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.2.keytab
--principal=kudu/127.22.37.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:43723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:35:51.262081 25304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:35:51.262495 25304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:35:51.262671 25304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:35:51.270104 25304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:35:51.270236 25304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:35:51.270406 25304 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.37.2
I20260620 18:35:51.280186 25304 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:43723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980527431848-22676-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980527431848-22676-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.2.keytab
--principal=kudu/127.22.37.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.37.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980527431848-22676-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.37.2
--webserver_port=0
--tserver_master_addrs=127.22.37.62:35633
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.37.2
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980527431848-22676-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:35:51.281592 25304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:35:51.282979 25304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:35:51.296190 25309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:51.296119 25310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:51.296152 25312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:35:51.296885 25304 server_base.cc:1034] running on GCE node
I20260620 18:35:51.297520 25304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:35:51.298475 25304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:35:51.299676 25304 hybrid_clock.cc:648] HybridClock initialized: now 1781980551299630 us; error 28 us; skew 500 ppm
I20260620 18:35:51.300015 25304 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:35:51 dist-test-slave-5lgp krb5kdc[25064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980551, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:35:51.307271 25304 init.cc:377] Logged in from keytab as kudu/127.22.37.2@KRBTEST.COM (short username kudu)
I20260620 18:35:51.308696 25304 webserver.cc:460] Webserver started at http://127.22.37.2:45565/ using document root <none> and password file <none>
I20260620 18:35:51.309443 25304 fs_manager.cc:359] Metadata directory not provided
I20260620 18:35:51.309585 25304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:35:51.309899 25304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:35:51.312741 25304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980527431848-22676-0/minicluster-data/ts-1/data/instance:
uuid: "e2cee904a8954f8b97d8ab4759360085"
format_stamp: "Formatted at 2026-06-20 18:35:51 on dist-test-slave-5lgp"
server_key: "c8a1d0e817289994ca69bd620766fdc2"
server_key_iv: "be20be1a9c59dd56c53c375db042e381"
server_key_version: "encryptionkey@0"
I20260620 18:35:51.313573 25304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980527431848-22676-0/minicluster-data/ts-1/wal/instance:
uuid: "e2cee904a8954f8b97d8ab4759360085"
format_stamp: "Formatted at 2026-06-20 18:35:51 on dist-test-slave-5lgp"
server_key: "c8a1d0e817289994ca69bd620766fdc2"
server_key_iv: "be20be1a9c59dd56c53c375db042e381"
server_key_version: "encryptionkey@0"
I20260620 18:35:51.319235 25304 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 18:35:51.323402 25319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:51.324934 25304 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 18:35:51.325094 25304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980527431848-22676-0/minicluster-data/ts-1/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980527431848-22676-0/minicluster-data/ts-1/wal
uuid: "e2cee904a8954f8b97d8ab4759360085"
format_stamp: "Formatted at 2026-06-20 18:35:51 on dist-test-slave-5lgp"
server_key: "c8a1d0e817289994ca69bd620766fdc2"
server_key_iv: "be20be1a9c59dd56c53c375db042e381"
server_key_version: "encryptionkey@0"
I20260620 18:35:51.325598 25304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980527431848-22676-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980527431848-22676-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980527431848-22676-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:35:51.354913 25304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:35:51.358829 25304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:35:51.359131 25304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:35:51.360098 25304 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:35:51.361543 25304 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:35:51.361614 25304 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:51.361696 25304 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:35:51.361740 25304 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:51.394488 25304 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.2:37473
I20260620 18:35:51.394565 25431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.2:37473 every 8 connection(s)
I20260620 18:35:51.395820 25304 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980527431848-22676-0/minicluster-data/ts-1/data/info.pb
I20260620 18:35:51.399732 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 25304
I20260620 18:35:51.399912 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980527431848-22676-0/minicluster-data/ts-1/wal/instance
I20260620 18:35:51.400391 22676 external_mini_cluster.cc:1428] Setting key e28bfac23d02b3bee04397482d4cd7e8
Jun 20 18:35:51 dist-test-slave-5lgp krb5kdc[25064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980551, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.2@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM
I20260620 18:35:51.418951 25300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:51.398356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.2:46667 (local address 127.22.37.62:35633)
0620 18:35:51.398557 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:51.398561 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:51.399796 (+ 1235us) server_negotiation.cc:407] Connection header received
0620 18:35:51.401813 (+ 2017us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:51.401819 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:51.401939 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:51.402042 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:51.404433 (+ 2391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:51.405138 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:51.407718 (+ 2580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:51.408476 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:51.408614 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:51.411960 (+ 3346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:51.411969 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:51.411977 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:51.412023 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:51.415697 (+ 3674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:51.416426 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:51.416433 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:51.416440 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:51.416619 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:51.417037 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:51.417044 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:51.417050 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:51.417466 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:51.417626 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:51.418292 (+ 666us) server_negotiation.cc:299] Negotiation successful
0620 18:35:51.418549 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
WARNING: no policy specified for kudu/127.22.37.3@KRBTEST.COM; defaulting to no policy
I20260620 18:35:51.419696 25434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:51.398830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:35633 (local address 127.22.37.2:46667)
0620 18:35:51.399623 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:51.399670 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:51.401558 (+ 1888us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:51.402581 (+ 1023us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:51.402592 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:51.403365 (+ 773us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:51.404224 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:51.404253 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:51.405305 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:51.405313 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:51.407536 (+ 2223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:51.407555 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:51.408702 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:51.408707 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:51.408907 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:51.409642 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:35:51.409664 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:51.411743 (+ 2079us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:35:51.415847 (+ 4104us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:51.415855 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:51.415869 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:51.416289 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:51.416739 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:51.416743 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:51.416745 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:51.416915 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:51.417589 (+ 674us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:35:51.417594 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:35:51.417912 (+ 318us) client_negotiation.cc:769] Sending connection context
0620 18:35:51.418143 (+ 231us) client_negotiation.cc:240] Negotiation successful
0620 18:35:51.418523 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":280,"threads_started":1}
I20260620 18:35:51.421242 25432 heartbeater.cc:347] Connected to a master server at 127.22.37.62:35633
I20260620 18:35:51.421593 25432 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.37.3@KRBTEST.COM" created.
I20260620 18:35:51.422340 25432 heartbeater.cc:511] Master 127.22.37.62:35633 requested a full tablet report, sending...
I20260620 18:35:51.423781 25109 ts_manager.cc:194] Registered new tserver with Master: e2cee904a8954f8b97d8ab4759360085 (127.22.37.2:37473)
I20260620 18:35:51.424597 25109 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.37.2@KRBTEST.COM'} at 127.22.37.2:46667
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.37.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.3.keytab.
Entry for principal kudu/127.22.37.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.3.keytab.
I20260620 18:35:51.445230 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980527431848-22676-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980527431848-22676-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980527431848-22676-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980527431848-22676-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.37.3:0
--local_ip_for_outbound_sockets=127.22.37.3
--webserver_interface=127.22.37.3
--webserver_port=0
--tserver_master_addrs=127.22.37.62:35633
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.3.keytab
--principal=kudu/127.22.37.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:43723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:35:51.764166 25437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:35:51.764575 25437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:35:51.764758 25437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:35:51.772248 25437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:35:51.772397 25437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:35:51.772583 25437 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.37.3
I20260620 18:35:51.782768 25437 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:43723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980527431848-22676-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980527431848-22676-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.3.keytab
--principal=kudu/127.22.37.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.37.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980527431848-22676-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.37.3
--webserver_port=0
--tserver_master_addrs=127.22.37.62:35633
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.37.3
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980527431848-22676-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:35:51.784224 25437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:35:51.785882 25437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:35:51.799906 25445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:51.800062 25443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:51.800408 25442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:35:51.800863 25437 server_base.cc:1034] running on GCE node
I20260620 18:35:51.801590 25437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:35:51.802506 25437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:35:51.803699 25437 hybrid_clock.cc:648] HybridClock initialized: now 1781980551803643 us; error 51 us; skew 500 ppm
I20260620 18:35:51.804009 25437 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:35:51 dist-test-slave-5lgp krb5kdc[25064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980551, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:35:51.807806 25437 init.cc:377] Logged in from keytab as kudu/127.22.37.3@KRBTEST.COM (short username kudu)
I20260620 18:35:51.809232 25437 webserver.cc:460] Webserver started at http://127.22.37.3:37299/ using document root <none> and password file <none>
I20260620 18:35:51.810096 25437 fs_manager.cc:359] Metadata directory not provided
I20260620 18:35:51.810225 25437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:35:51.810551 25437 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:35:51.813723 25437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980527431848-22676-0/minicluster-data/ts-2/data/instance:
uuid: "d41fac0edea04c7eb83b3c6df6d975f1"
format_stamp: "Formatted at 2026-06-20 18:35:51 on dist-test-slave-5lgp"
server_key: "e635d710038897fee0f9cebc36e4ee10"
server_key_iv: "48309709a37964f3b4c4bbefd3ea3d3f"
server_key_version: "encryptionkey@0"
I20260620 18:35:51.814563 25437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980527431848-22676-0/minicluster-data/ts-2/wal/instance:
uuid: "d41fac0edea04c7eb83b3c6df6d975f1"
format_stamp: "Formatted at 2026-06-20 18:35:51 on dist-test-slave-5lgp"
server_key: "e635d710038897fee0f9cebc36e4ee10"
server_key_iv: "48309709a37964f3b4c4bbefd3ea3d3f"
server_key_version: "encryptionkey@0"
I20260620 18:35:51.820461 25437 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260620 18:35:51.823968 25452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:51.825285 25437 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:35:51.825474 25437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980527431848-22676-0/minicluster-data/ts-2/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980527431848-22676-0/minicluster-data/ts-2/wal
uuid: "d41fac0edea04c7eb83b3c6df6d975f1"
format_stamp: "Formatted at 2026-06-20 18:35:51 on dist-test-slave-5lgp"
server_key: "e635d710038897fee0f9cebc36e4ee10"
server_key_iv: "48309709a37964f3b4c4bbefd3ea3d3f"
server_key_version: "encryptionkey@0"
I20260620 18:35:51.825681 25437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980527431848-22676-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980527431848-22676-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980527431848-22676-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:35:51.842953 25437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:35:51.846822 25437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:35:51.847112 25437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:35:51.848080 25437 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:35:51.849457 25437 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:35:51.849537 25437 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:51.849705 25437 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:35:51.849754 25437 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:51.883579 25437 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.3:40147
I20260620 18:35:51.883590 25564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.3:40147 every 8 connection(s)
I20260620 18:35:51.884900 25437 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980527431848-22676-0/minicluster-data/ts-2/data/info.pb
I20260620 18:35:51.892987 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 25437
I20260620 18:35:51.893187 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980527431848-22676-0/minicluster-data/ts-2/wal/instance
I20260620 18:35:51.893684 22676 external_mini_cluster.cc:1428] Setting key cc1ffd3a29a2bdd4cad3e4961ccec43a
Jun 20 18:35:51 dist-test-slave-5lgp krb5kdc[25064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980551, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.3@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM
I20260620 18:35:51.905459 25300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:51.887479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.3:45817 (local address 127.22.37.62:35633)
0620 18:35:51.887690 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:51.887697 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:51.888940 (+ 1243us) server_negotiation.cc:407] Connection header received
0620 18:35:51.890842 (+ 1902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:51.890848 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:51.890955 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:51.891074 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:51.892573 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:51.893496 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:51.895427 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:51.896073 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:51.896201 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:51.899829 (+ 3628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:51.899840 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:51.899843 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:51.899876 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:51.902440 (+ 2564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:51.903183 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:51.903196 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:51.903198 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:51.903322 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:51.903736 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:51.903743 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:51.903747 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:51.904009 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:51.904161 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:51.904855 (+ 694us) server_negotiation.cc:299] Negotiation successful
0620 18:35:51.905070 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 18:35:51.906477 25567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:51.888081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:35633 (local address 127.22.37.3:45817)
0620 18:35:51.888764 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:51.888805 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:51.890562 (+ 1757us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:51.891309 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:51.891317 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:51.891831 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:51.892397 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:51.892413 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:51.893662 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:51.893666 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:51.895283 (+ 1617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:51.895294 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:51.896271 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:51.896276 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:51.896489 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:51.897234 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:35:51.897257 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:51.899631 (+ 2374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:35:51.902597 (+ 2966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:51.902606 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:51.902624 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:51.903056 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:51.903433 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:51.903438 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:51.903442 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:51.903627 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:51.904132 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:35:51.904140 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:35:51.904494 (+ 354us) client_negotiation.cc:769] Sending connection context
0620 18:35:51.904780 (+ 286us) client_negotiation.cc:240] Negotiation successful
0620 18:35:51.905321 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":271,"threads_started":1}
I20260620 18:35:51.908104 25565 heartbeater.cc:347] Connected to a master server at 127.22.37.62:35633
I20260620 18:35:51.908500 25565 heartbeater.cc:464] Registering TS with master...
I20260620 18:35:51.909497 25565 heartbeater.cc:511] Master 127.22.37.62:35633 requested a full tablet report, sending...
I20260620 18:35:51.910858 25109 ts_manager.cc:194] Registered new tserver with Master: d41fac0edea04c7eb83b3c6df6d975f1 (127.22.37.3:40147)
I20260620 18:35:51.911648 25109 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.37.3@KRBTEST.COM'} at 127.22.37.3:45817
I20260620 18:35:51.921509 22676 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:35:51.922668 25298 heartbeater.cc:503] Master 127.22.37.62:35633 was elected leader, sending a full tablet report...
W20260620 18:35:51.951613 25300 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:35:51.949252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36072 (local address 127.22.37.62:35633)
0620 18:35:51.949525 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:51.949535 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:51.949873 (+ 338us) server_negotiation.cc:407] Connection header received
0620 18:35:51.950015 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:51.950021 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:51.950085 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:51.950172 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:51.951335 (+ 1163us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36072: BlockingRecv error: recv got EOF from 127.0.0.1:36072 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 18:35:51.955310 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 25170
W20260620 18:35:51.958036 22676 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 25170: Not authorized: Client connection negotiation failed: client connection to 127.22.37.1:40971: server requires authentication, but client does not have Kerberos credentials available
I20260620 18:35:51.958225 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 25170
I20260620 18:35:51.975338 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 25304
W20260620 18:35:51.977867 22676 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 25304: Not authorized: Client connection negotiation failed: client connection to 127.22.37.2:37473: server requires authentication, but client does not have Kerberos credentials available
I20260620 18:35:51.978067 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 25304
I20260620 18:35:51.993741 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 25437
W20260620 18:35:51.996603 22676 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 25437: Not authorized: Client connection negotiation failed: client connection to 127.22.37.3:40147: server requires authentication, but client does not have Kerberos credentials available
I20260620 18:35:51.996748 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 25437
I20260620 18:35:52.012562 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 25077
I20260620 18:35:52.013945 25139 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:35:52.117022 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 25077
2026-06-20T18:35:52Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3488 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980527431848-22676-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:35:53 dist-test-slave-5lgp krb5kdc[25582](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:35:53 dist-test-slave-5lgp krb5kdc[25582](info): set up 2 sockets
Jun 20 18:35:53 dist-test-slave-5lgp krb5kdc[25582](info): commencing operation
krb5kdc: starting...
W20260620 18:35:54.589731 22676 mini_kdc.cc:121] Time spent starting KDC: real 2.429s user 0.004s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980527431848-22676-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980527431848-22676-0/krb5kdc/test-user.keytab.
Jun 20 18:35:54 dist-test-slave-5lgp krb5kdc[25582](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980554, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:35:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:35:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.37.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.37.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.37.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab.
Entry for principal kudu/127.22.37.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab.
I20260620 18:35:54.780618 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980527431848-22676-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980527431848-22676-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980527431848-22676-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.62:33001
--webserver_interface=127.22.37.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab
--principal=kudu/127.22.37.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:43167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.37.62:33001
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:35:55.096302 25595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:35:55.096599 25595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:35:55.096832 25595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:35:55.104287 25595 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:35:55.104398 25595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:35:55.104645 25595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:35:55.104786 25595 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:35:55.104864 25595 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:35:55.115010 25595 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:43167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980527431848-22676-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980527431848-22676-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.37.62:33001
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab
--principal=kudu/127.22.37.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.62:33001
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.37.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980527431848-22676-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:35:55.116339 25595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:35:55.117951 25595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:35:55.130752 25600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:55.130766 25601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:55.130681 25603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:35:55.131630 25595 server_base.cc:1034] running on GCE node
I20260620 18:35:55.132387 25595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:35:55.133698 25595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:35:55.134902 25595 hybrid_clock.cc:648] HybridClock initialized: now 1781980555134859 us; error 28 us; skew 500 ppm
I20260620 18:35:55.135190 25595 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:35:55 dist-test-slave-5lgp krb5kdc[25582](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980555, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:35:55.138623 25595 init.cc:377] Logged in from keytab as kudu/127.22.37.62@KRBTEST.COM (short username kudu)
I20260620 18:35:55.140861 25595 webserver.cc:460] Webserver started at http://127.22.37.62:46335/ using document root <none> and password file <none>
I20260620 18:35:55.141870 25595 fs_manager.cc:359] Metadata directory not provided
I20260620 18:35:55.142022 25595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:35:55.142374 25595 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:35:55.146090 25595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980527431848-22676-0/minicluster-data/master-0/data/instance:
uuid: "7ec6003ba95243a0a2e59255b69c8b54"
format_stamp: "Formatted at 2026-06-20 18:35:55 on dist-test-slave-5lgp"
server_key: "99f42fdcd41e9f3b9c8e877b67c55550"
server_key_iv: "d2358f8e25035e9fdd26eb7f78c66b6b"
server_key_version: "encryptionkey@0"
I20260620 18:35:55.146868 25595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980527431848-22676-0/minicluster-data/master-0/wal/instance:
uuid: "7ec6003ba95243a0a2e59255b69c8b54"
format_stamp: "Formatted at 2026-06-20 18:35:55 on dist-test-slave-5lgp"
server_key: "99f42fdcd41e9f3b9c8e877b67c55550"
server_key_iv: "d2358f8e25035e9fdd26eb7f78c66b6b"
server_key_version: "encryptionkey@0"
I20260620 18:35:55.152388 25595 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 18:35:55.156080 25610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:55.157507 25595 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 18:35:55.157676 25595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980527431848-22676-0/minicluster-data/master-0/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980527431848-22676-0/minicluster-data/master-0/wal
uuid: "7ec6003ba95243a0a2e59255b69c8b54"
format_stamp: "Formatted at 2026-06-20 18:35:55 on dist-test-slave-5lgp"
server_key: "99f42fdcd41e9f3b9c8e877b67c55550"
server_key_iv: "d2358f8e25035e9fdd26eb7f78c66b6b"
server_key_version: "encryptionkey@0"
I20260620 18:35:55.157882 25595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980527431848-22676-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980527431848-22676-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980527431848-22676-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:35:55.182392 25595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:35:55.187070 25595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:35:55.187335 25595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:35:55.206240 25595 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.62:33001
I20260620 18:35:55.206292 25661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.62:33001 every 8 connection(s)
I20260620 18:35:55.207638 25595 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
I20260620 18:35:55.208993 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 25595
I20260620 18:35:55.209223 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980527431848-22676-0/minicluster-data/master-0/wal/instance
I20260620 18:35:55.210055 22676 external_mini_cluster.cc:1428] Setting key b3de05f6fe34b511b6a4ad514def7f7a
I20260620 18:35:55.212620 25662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 18:35:55 dist-test-slave-5lgp krb5kdc[25582](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980554, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM
I20260620 18:35:55.226140 25662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ec6003ba95243a0a2e59255b69c8b54: Bootstrap starting.
I20260620 18:35:55.229338 25662 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ec6003ba95243a0a2e59255b69c8b54: Neither blocks nor log segments found. Creating new log.
I20260620 18:35:55.230393 25662 log.cc:824] T 00000000000000000000000000000000 P 7ec6003ba95243a0a2e59255b69c8b54: Log is configured to *not* fsync() on all Append() calls
I20260620 18:35:55.233247 25665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:55.212023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58164 (local address 127.22.37.62:33001)
0620 18:35:55.212876 (+ 853us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:55.212889 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:55.212935 (+ 46us) server_negotiation.cc:407] Connection header received
0620 18:35:55.214653 (+ 1718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:55.214676 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:55.215099 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:55.215460 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:55.217353 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:55.218768 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:55.220559 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:55.221236 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:55.221438 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:55.226025 (+ 4587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:55.226033 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:55.226049 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:55.226084 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:55.229119 (+ 3035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:55.229929 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:55.229944 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:55.229951 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:55.230122 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:55.230511 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:55.230516 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:55.230520 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:55.231264 (+ 744us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:55.231597 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:55.231900 (+ 303us) server_negotiation.cc:299] Negotiation successful
0620 18:35:55.232268 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":373,"threads_started":1}
I20260620 18:35:55.233541 25662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ec6003ba95243a0a2e59255b69c8b54: No bootstrap required, opened a new log
I20260620 18:35:55.237169 22676 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:35:55.241199 25662 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ec6003ba95243a0a2e59255b69c8b54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ec6003ba95243a0a2e59255b69c8b54" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 33001 } }
I20260620 18:35:55.241631 25662 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7ec6003ba95243a0a2e59255b69c8b54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:35:55.241731 25662 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ec6003ba95243a0a2e59255b69c8b54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ec6003ba95243a0a2e59255b69c8b54, State: Initialized, Role: FOLLOWER
I20260620 18:35:55.242269 25662 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ec6003ba95243a0a2e59255b69c8b54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ec6003ba95243a0a2e59255b69c8b54" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 33001 } }
I20260620 18:35:55.242514 25662 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7ec6003ba95243a0a2e59255b69c8b54 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:35:55.242645 25662 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ec6003ba95243a0a2e59255b69c8b54 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:35:55.242818 25662 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ec6003ba95243a0a2e59255b69c8b54 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:35:55.244871 25662 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ec6003ba95243a0a2e59255b69c8b54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ec6003ba95243a0a2e59255b69c8b54" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 33001 } }
I20260620 18:35:55.245332 25662 leader_election.cc:302] T 00000000000000000000000000000000 P 7ec6003ba95243a0a2e59255b69c8b54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7ec6003ba95243a0a2e59255b69c8b54; no voters:
I20260620 18:35:55.245990 25662 leader_election.cc:288] T 00000000000000000000000000000000 P 7ec6003ba95243a0a2e59255b69c8b54 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:35:55.246088 25668 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7ec6003ba95243a0a2e59255b69c8b54 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:35:55.247222 25668 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7ec6003ba95243a0a2e59255b69c8b54 [term 1 LEADER]: Becoming Leader. State: Replica: 7ec6003ba95243a0a2e59255b69c8b54, State: Running, Role: LEADER
I20260620 18:35:55.247843 25668 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ec6003ba95243a0a2e59255b69c8b54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ec6003ba95243a0a2e59255b69c8b54" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 33001 } }
I20260620 18:35:55.248338 25662 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ec6003ba95243a0a2e59255b69c8b54 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:35:55.250674 25670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ec6003ba95243a0a2e59255b69c8b54 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ec6003ba95243a0a2e59255b69c8b54. Latest consensus state: current_term: 1 leader_uuid: "7ec6003ba95243a0a2e59255b69c8b54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ec6003ba95243a0a2e59255b69c8b54" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 33001 } } }
I20260620 18:35:55.251099 25670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ec6003ba95243a0a2e59255b69c8b54 [sys.catalog]: This master's current role is: LEADER
I20260620 18:35:55.251741 25669 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ec6003ba95243a0a2e59255b69c8b54 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7ec6003ba95243a0a2e59255b69c8b54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ec6003ba95243a0a2e59255b69c8b54" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 33001 } } }
I20260620 18:35:55.252063 25669 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ec6003ba95243a0a2e59255b69c8b54 [sys.catalog]: This master's current role is: LEADER
I20260620 18:35:55.259907 25673 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:35:55.266455 25673 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:35:55.279237 25673 catalog_manager.cc:1347] Generated new cluster ID: 6fb6cb45a4c8487dadc452bf58fdf7e8
I20260620 18:35:55.279366 25673 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:35:55.296343 25665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:55.287306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58172 (local address 127.22.37.62:33001)
0620 18:35:55.287494 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:55.287500 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:55.288072 (+ 572us) server_negotiation.cc:407] Connection header received
0620 18:35:55.288267 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:55.288272 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:55.288348 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:55.288449 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:55.289683 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:55.290825 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:55.292706 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:55.293335 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:55.293452 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:55.294660 (+ 1208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:55.294666 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:55.294671 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:35:55.295492 (+ 821us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:55.295565 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:35:55.295599 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:55.295713 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:55.295873 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 18:35:55.295965 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 18:35:55.298656 25673 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:35:55.299863 25673 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:35:55.314549 25673 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7ec6003ba95243a0a2e59255b69c8b54: Generated new TSK 0
I20260620 18:35:55.315809 25673 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:35:55.325354 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 25595
I20260620 18:35:55.327204 25657 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:35:55.417870 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 25595
2026-06-20T18:35:55Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (3293 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980527431848-22676-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:35:55 dist-test-slave-5lgp krb5kdc[25697](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:35:55 dist-test-slave-5lgp krb5kdc[25697](info): set up 2 sockets
Jun 20 18:35:55 dist-test-slave-5lgp krb5kdc[25697](info): commencing operation
krb5kdc: starting...
W20260620 18:35:56.501704 22676 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.004s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980527431848-22676-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980527431848-22676-0/krb5kdc/test-user.keytab.
Jun 20 18:35:56 dist-test-slave-5lgp krb5kdc[25697](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980556, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:35:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:35:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.37.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.37.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.37.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab.
Entry for principal kudu/127.22.37.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab.
I20260620 18:35:56.687003 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980527431848-22676-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980527431848-22676-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980527431848-22676-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.62:33321
--webserver_interface=127.22.37.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab
--principal=kudu/127.22.37.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:45769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.37.62:33321
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:35:57.001755 25710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:35:57.002090 25710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:35:57.002264 25710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:35:57.009678 25710 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:35:57.009825 25710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:35:57.009938 25710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:35:57.010039 25710 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:35:57.010128 25710 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:35:57.019969 25710 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:45769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980527431848-22676-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980527431848-22676-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.37.62:33321
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab
--principal=kudu/127.22.37.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.62:33321
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.37.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980527431848-22676-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:35:57.021339 25710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:35:57.022677 25710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:35:57.034320 25715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:57.034341 25718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:57.034516 25716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:35:57.035218 25710 server_base.cc:1034] running on GCE node
I20260620 18:35:57.035991 25710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:35:57.037348 25710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:35:57.038617 25710 hybrid_clock.cc:648] HybridClock initialized: now 1781980557038583 us; error 47 us; skew 500 ppm
I20260620 18:35:57.038939 25710 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:35:57 dist-test-slave-5lgp krb5kdc[25697](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980557, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:35:57.046428 25710 init.cc:377] Logged in from keytab as kudu/127.22.37.62@KRBTEST.COM (short username kudu)
I20260620 18:35:57.047848 25710 webserver.cc:460] Webserver started at http://127.22.37.62:45289/ using document root <none> and password file <none>
I20260620 18:35:57.048591 25710 fs_manager.cc:359] Metadata directory not provided
I20260620 18:35:57.048736 25710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:35:57.049052 25710 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:35:57.051997 25710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980527431848-22676-0/minicluster-data/master-0/data/instance:
uuid: "0b2464db633b4f1f9e27c6625bab05ec"
format_stamp: "Formatted at 2026-06-20 18:35:57 on dist-test-slave-5lgp"
server_key: "a7ce1be5eed0846343d871572c816645"
server_key_iv: "c42841401a430836f9e769b91ab926bf"
server_key_version: "encryptionkey@0"
I20260620 18:35:57.052748 25710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980527431848-22676-0/minicluster-data/master-0/wal/instance:
uuid: "0b2464db633b4f1f9e27c6625bab05ec"
format_stamp: "Formatted at 2026-06-20 18:35:57 on dist-test-slave-5lgp"
server_key: "a7ce1be5eed0846343d871572c816645"
server_key_iv: "c42841401a430836f9e769b91ab926bf"
server_key_version: "encryptionkey@0"
I20260620 18:35:57.057641 25710 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 18:35:57.060927 25725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:57.062275 25710 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:35:57.062454 25710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980527431848-22676-0/minicluster-data/master-0/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980527431848-22676-0/minicluster-data/master-0/wal
uuid: "0b2464db633b4f1f9e27c6625bab05ec"
format_stamp: "Formatted at 2026-06-20 18:35:57 on dist-test-slave-5lgp"
server_key: "a7ce1be5eed0846343d871572c816645"
server_key_iv: "c42841401a430836f9e769b91ab926bf"
server_key_version: "encryptionkey@0"
I20260620 18:35:57.062642 25710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980527431848-22676-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980527431848-22676-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980527431848-22676-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:35:57.075699 25710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:35:57.079476 25710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:35:57.079758 25710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:35:57.097319 25710 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.62:33321
I20260620 18:35:57.097353 25776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.62:33321 every 8 connection(s)
I20260620 18:35:57.098690 25710 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
I20260620 18:35:57.102348 25777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:35:57.103479 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 25710
I20260620 18:35:57.103714 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980527431848-22676-0/minicluster-data/master-0/wal/instance
I20260620 18:35:57.104550 22676 external_mini_cluster.cc:1428] Setting key 8de431cfc4faae4969f25b7d06ab4c6f
I20260620 18:35:57.115003 25777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b2464db633b4f1f9e27c6625bab05ec: Bootstrap starting.
Jun 20 18:35:57 dist-test-slave-5lgp krb5kdc[25697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980556, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM
I20260620 18:35:57.118562 25777 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b2464db633b4f1f9e27c6625bab05ec: Neither blocks nor log segments found. Creating new log.
I20260620 18:35:57.119608 25777 log.cc:824] T 00000000000000000000000000000000 P 0b2464db633b4f1f9e27c6625bab05ec: Log is configured to *not* fsync() on all Append() calls
I20260620 18:35:57.122510 25777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b2464db633b4f1f9e27c6625bab05ec: No bootstrap required, opened a new log
I20260620 18:35:57.126042 25780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:57.106673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47820 (local address 127.22.37.62:33321)
0620 18:35:57.107402 (+ 729us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:57.107418 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:57.107460 (+ 42us) server_negotiation.cc:407] Connection header received
0620 18:35:57.109034 (+ 1574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:57.109067 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:57.109507 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:57.109897 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:57.111236 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:57.112494 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:57.113782 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:57.114435 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:57.114637 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:57.119063 (+ 4426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:57.119070 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:57.119085 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:57.119117 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:57.122147 (+ 3030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:57.122839 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:57.122857 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:57.122862 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:57.123027 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:57.123392 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:57.123398 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:57.123401 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:57.123898 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:57.124261 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:57.124583 (+ 322us) server_negotiation.cc:299] Negotiation successful
0620 18:35:57.125002 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":324,"threads_started":1}
I20260620 18:35:57.130271 25777 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b2464db633b4f1f9e27c6625bab05ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b2464db633b4f1f9e27c6625bab05ec" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 33321 } }
I20260620 18:35:57.130646 25777 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0b2464db633b4f1f9e27c6625bab05ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:35:57.130752 25777 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b2464db633b4f1f9e27c6625bab05ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b2464db633b4f1f9e27c6625bab05ec, State: Initialized, Role: FOLLOWER
I20260620 18:35:57.131313 25777 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b2464db633b4f1f9e27c6625bab05ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b2464db633b4f1f9e27c6625bab05ec" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 33321 } }
I20260620 18:35:57.131551 25777 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0b2464db633b4f1f9e27c6625bab05ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:35:57.131692 25777 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b2464db633b4f1f9e27c6625bab05ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:35:57.131855 25777 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b2464db633b4f1f9e27c6625bab05ec [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:35:57.134286 25777 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b2464db633b4f1f9e27c6625bab05ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b2464db633b4f1f9e27c6625bab05ec" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 33321 } }
I20260620 18:35:57.134706 25777 leader_election.cc:302] T 00000000000000000000000000000000 P 0b2464db633b4f1f9e27c6625bab05ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b2464db633b4f1f9e27c6625bab05ec; no voters:
I20260620 18:35:57.135196 25777 leader_election.cc:288] T 00000000000000000000000000000000 P 0b2464db633b4f1f9e27c6625bab05ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:35:57.135366 25782 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b2464db633b4f1f9e27c6625bab05ec [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:35:57.136034 25782 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b2464db633b4f1f9e27c6625bab05ec [term 1 LEADER]: Becoming Leader. State: Replica: 0b2464db633b4f1f9e27c6625bab05ec, State: Running, Role: LEADER
I20260620 18:35:57.137734 25782 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b2464db633b4f1f9e27c6625bab05ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b2464db633b4f1f9e27c6625bab05ec" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 33321 } }
I20260620 18:35:57.137105 25777 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b2464db633b4f1f9e27c6625bab05ec [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:35:57.141835 25784 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b2464db633b4f1f9e27c6625bab05ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b2464db633b4f1f9e27c6625bab05ec. Latest consensus state: current_term: 1 leader_uuid: "0b2464db633b4f1f9e27c6625bab05ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b2464db633b4f1f9e27c6625bab05ec" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 33321 } } }
I20260620 18:35:57.142203 25784 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b2464db633b4f1f9e27c6625bab05ec [sys.catalog]: This master's current role is: LEADER
I20260620 18:35:57.143299 25783 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b2464db633b4f1f9e27c6625bab05ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0b2464db633b4f1f9e27c6625bab05ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b2464db633b4f1f9e27c6625bab05ec" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 33321 } } }
I20260620 18:35:57.146293 25783 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b2464db633b4f1f9e27c6625bab05ec [sys.catalog]: This master's current role is: LEADER
I20260620 18:35:57.148025 25788 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:35:57.153304 25788 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:35:57.161901 25788 catalog_manager.cc:1347] Generated new cluster ID: c0e3e1733913464696256699bc22da0e
I20260620 18:35:57.162019 25788 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:35:57.177452 25788 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:35:57.178759 25788 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:35:57.187475 25788 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0b2464db633b4f1f9e27c6625bab05ec: Generated new TSK 0
I20260620 18:35:57.188323 25788 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.37.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.37.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.37.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.1.keytab.
Entry for principal kudu/127.22.37.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.1.keytab.
I20260620 18:35:57.248497 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980527431848-22676-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980527431848-22676-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980527431848-22676-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980527431848-22676-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.37.1:0
--local_ip_for_outbound_sockets=127.22.37.1
--webserver_interface=127.22.37.1
--webserver_port=0
--tserver_master_addrs=127.22.37.62:33321
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.1.keytab
--principal=kudu/127.22.37.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:45769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:35:57.572278 25803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:35:57.572554 25803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:35:57.572702 25803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:35:57.580049 25803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:35:57.580181 25803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:35:57.580368 25803 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.37.1
I20260620 18:35:57.590101 25803 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:45769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980527431848-22676-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980527431848-22676-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.1.keytab
--principal=kudu/127.22.37.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.37.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980527431848-22676-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.37.1
--webserver_port=0
--tserver_master_addrs=127.22.37.62:33321
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.37.1
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980527431848-22676-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:35:57.591413 25803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:35:57.592741 25803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:35:57.606003 25809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:57.606082 25811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:57.606031 25808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:35:57.606959 25803 server_base.cc:1034] running on GCE node
I20260620 18:35:57.607633 25803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:35:57.608640 25803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:35:57.609891 25803 hybrid_clock.cc:648] HybridClock initialized: now 1781980557609822 us; error 58 us; skew 500 ppm
I20260620 18:35:57.610265 25803 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:35:57 dist-test-slave-5lgp krb5kdc[25697](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980557, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:35:57.619306 25803 init.cc:377] Logged in from keytab as kudu/127.22.37.1@KRBTEST.COM (short username kudu)
I20260620 18:35:57.620942 25803 webserver.cc:460] Webserver started at http://127.22.37.1:39115/ using document root <none> and password file <none>
I20260620 18:35:57.621767 25803 fs_manager.cc:359] Metadata directory not provided
I20260620 18:35:57.621915 25803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:35:57.622221 25803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:35:57.625711 25803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980527431848-22676-0/minicluster-data/ts-0/data/instance:
uuid: "bd6106d027f34a8fb464eaa3e04030c2"
format_stamp: "Formatted at 2026-06-20 18:35:57 on dist-test-slave-5lgp"
server_key: "7bd1f0d65a0e50171b818ecfa9c32725"
server_key_iv: "f407a56ecf66200788f418dbae4c4288"
server_key_version: "encryptionkey@0"
I20260620 18:35:57.626508 25803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980527431848-22676-0/minicluster-data/ts-0/wal/instance:
uuid: "bd6106d027f34a8fb464eaa3e04030c2"
format_stamp: "Formatted at 2026-06-20 18:35:57 on dist-test-slave-5lgp"
server_key: "7bd1f0d65a0e50171b818ecfa9c32725"
server_key_iv: "f407a56ecf66200788f418dbae4c4288"
server_key_version: "encryptionkey@0"
I20260620 18:35:57.631801 25803 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260620 18:35:57.635519 25818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:57.636931 25803 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:35:57.637147 25803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980527431848-22676-0/minicluster-data/ts-0/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980527431848-22676-0/minicluster-data/ts-0/wal
uuid: "bd6106d027f34a8fb464eaa3e04030c2"
format_stamp: "Formatted at 2026-06-20 18:35:57 on dist-test-slave-5lgp"
server_key: "7bd1f0d65a0e50171b818ecfa9c32725"
server_key_iv: "f407a56ecf66200788f418dbae4c4288"
server_key_version: "encryptionkey@0"
I20260620 18:35:57.637367 25803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980527431848-22676-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980527431848-22676-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980527431848-22676-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:35:57.660342 25803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:35:57.664877 25803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:35:57.665208 25803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:35:57.666327 25803 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:35:57.667724 25803 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:35:57.667826 25803 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:57.667975 25803 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:35:57.668046 25803 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:57.700237 25803 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.1:32991
I20260620 18:35:57.700284 25930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.1:32991 every 8 connection(s)
I20260620 18:35:57.701632 25803 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980527431848-22676-0/minicluster-data/ts-0/data/info.pb
I20260620 18:35:57.709127 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 25803
I20260620 18:35:57.709327 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980527431848-22676-0/minicluster-data/ts-0/wal/instance
I20260620 18:35:57.710139 22676 external_mini_cluster.cc:1428] Setting key 51fbdafc70247a3d31aba4e583e90d0f
Jun 20 18:35:57 dist-test-slave-5lgp krb5kdc[25697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980557, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.1@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM
I20260620 18:35:57.721936 25933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:57.704076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.1:57039 (local address 127.22.37.62:33321)
0620 18:35:57.704544 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:57.704550 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:57.705570 (+ 1020us) server_negotiation.cc:407] Connection header received
0620 18:35:57.707296 (+ 1726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:57.707300 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:57.707367 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:57.707464 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:57.708916 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:57.710118 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:57.711894 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:57.712455 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:57.712601 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:57.715880 (+ 3279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:57.715890 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:57.715897 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:57.715943 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:57.718667 (+ 2724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:57.719569 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:57.719573 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:57.719575 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:57.719683 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:57.720154 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:57.720157 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:57.720159 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:57.720397 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:57.720500 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:57.721353 (+ 853us) server_negotiation.cc:299] Negotiation successful
0620 18:35:57.721606 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":205,"threads_started":1}
I20260620 18:35:57.723320 25934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:57.704652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:33321 (local address 127.22.37.1:57039)
0620 18:35:57.705356 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:57.705462 (+ 106us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:57.707048 (+ 1586us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:57.707684 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:57.707693 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:57.708171 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:57.708724 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:57.708750 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:57.710326 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:57.710330 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:57.711727 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:57.711742 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:57.712678 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:57.712682 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:57.712863 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:57.713606 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:35:57.713626 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:57.715667 (+ 2041us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:35:57.718801 (+ 3134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:57.718812 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:57.718832 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:57.719459 (+ 627us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:57.719783 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:57.719790 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:57.719797 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:57.720040 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:57.720514 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:35:57.720525 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:35:57.720969 (+ 444us) client_negotiation.cc:769] Sending connection context
0620 18:35:57.721281 (+ 312us) client_negotiation.cc:240] Negotiation successful
0620 18:35:57.721867 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":285,"threads_started":1}
I20260620 18:35:57.725031 25931 heartbeater.cc:347] Connected to a master server at 127.22.37.62:33321
I20260620 18:35:57.725349 25931 heartbeater.cc:464] Registering TS with master...
I20260620 18:35:57.726110 25931 heartbeater.cc:511] Master 127.22.37.62:33321 requested a full tablet report, sending...
I20260620 18:35:57.728515 25742 ts_manager.cc:194] Registered new tserver with Master: bd6106d027f34a8fb464eaa3e04030c2 (127.22.37.1:32991)
WARNING: no policy specified for kudu/127.22.37.2@KRBTEST.COM; defaulting to no policy
I20260620 18:35:57.730219 25742 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.37.1@KRBTEST.COM'} at 127.22.37.1:57039
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.37.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.37.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.2.keytab.
Entry for principal kudu/127.22.37.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.2.keytab.
I20260620 18:35:57.756158 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980527431848-22676-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980527431848-22676-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980527431848-22676-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980527431848-22676-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.37.2:0
--local_ip_for_outbound_sockets=127.22.37.2
--webserver_interface=127.22.37.2
--webserver_port=0
--tserver_master_addrs=127.22.37.62:33321
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.2.keytab
--principal=kudu/127.22.37.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:45769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:35:58.089339 25937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:35:58.089776 25937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:35:58.090029 25937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:35:58.097327 25937 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:35:58.097503 25937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:35:58.097703 25937 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.37.2
I20260620 18:35:58.107353 25937 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:45769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980527431848-22676-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980527431848-22676-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.2.keytab
--principal=kudu/127.22.37.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.37.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980527431848-22676-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.37.2
--webserver_port=0
--tserver_master_addrs=127.22.37.62:33321
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.37.2
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980527431848-22676-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:35:58.108639 25937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:35:58.109997 25937 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:35:58.122727 25942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:58.122685 25945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:58.122772 25943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:35:58.129091 25937 server_base.cc:1034] running on GCE node
I20260620 18:35:58.129722 25937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:35:58.130571 25937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:35:58.131742 25937 hybrid_clock.cc:648] HybridClock initialized: now 1781980558131685 us; error 53 us; skew 500 ppm
I20260620 18:35:58.132063 25937 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:35:58 dist-test-slave-5lgp krb5kdc[25697](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980558, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:35:58.135154 25937 init.cc:377] Logged in from keytab as kudu/127.22.37.2@KRBTEST.COM (short username kudu)
I20260620 18:35:58.136626 25937 webserver.cc:460] Webserver started at http://127.22.37.2:46863/ using document root <none> and password file <none>
I20260620 18:35:58.137305 25937 fs_manager.cc:359] Metadata directory not provided
I20260620 18:35:58.137463 25937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:35:58.137748 25937 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:35:58.140486 25937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980527431848-22676-0/minicluster-data/ts-1/data/instance:
uuid: "3fd51f0f60d847789b072436c34bcfa9"
format_stamp: "Formatted at 2026-06-20 18:35:58 on dist-test-slave-5lgp"
server_key: "d1e8beeff69483f5d2ea0b99ef0f2626"
server_key_iv: "b5f058b13483b84cc44a57d92c382a93"
server_key_version: "encryptionkey@0"
I20260620 18:35:58.141283 25937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980527431848-22676-0/minicluster-data/ts-1/wal/instance:
uuid: "3fd51f0f60d847789b072436c34bcfa9"
format_stamp: "Formatted at 2026-06-20 18:35:58 on dist-test-slave-5lgp"
server_key: "d1e8beeff69483f5d2ea0b99ef0f2626"
server_key_iv: "b5f058b13483b84cc44a57d92c382a93"
server_key_version: "encryptionkey@0"
I20260620 18:35:58.146142 25937 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 18:35:58.149447 25952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:58.150777 25937 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 18:35:58.150966 25937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980527431848-22676-0/minicluster-data/ts-1/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980527431848-22676-0/minicluster-data/ts-1/wal
uuid: "3fd51f0f60d847789b072436c34bcfa9"
format_stamp: "Formatted at 2026-06-20 18:35:58 on dist-test-slave-5lgp"
server_key: "d1e8beeff69483f5d2ea0b99ef0f2626"
server_key_iv: "b5f058b13483b84cc44a57d92c382a93"
server_key_version: "encryptionkey@0"
I20260620 18:35:58.151158 25937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980527431848-22676-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980527431848-22676-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980527431848-22676-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:35:58.171576 25937 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:35:58.175304 25937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:35:58.175580 25937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:35:58.176564 25937 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:35:58.177959 25937 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:35:58.178061 25937 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:58.178206 25937 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:35:58.178274 25937 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:58.211377 25937 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.2:42727
I20260620 18:35:58.211422 26064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.2:42727 every 8 connection(s)
I20260620 18:35:58.212638 25937 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980527431848-22676-0/minicluster-data/ts-1/data/info.pb
I20260620 18:35:58.219694 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 25937
I20260620 18:35:58.219874 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980527431848-22676-0/minicluster-data/ts-1/wal/instance
I20260620 18:35:58.220427 22676 external_mini_cluster.cc:1428] Setting key fbc294c5dcbea9dff8c021b3c5250c0c
Jun 20 18:35:58 dist-test-slave-5lgp krb5kdc[25697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980558, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.2@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM
I20260620 18:35:58.233224 25933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:58.215166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.2:57975 (local address 127.22.37.62:33321)
0620 18:35:58.215366 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:58.215372 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:58.216578 (+ 1206us) server_negotiation.cc:407] Connection header received
0620 18:35:58.218373 (+ 1795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:58.218377 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:58.218441 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:58.218526 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:58.220196 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:58.220976 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:58.223025 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:58.223431 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:58.223526 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:58.227621 (+ 4095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:58.227626 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:58.227629 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:58.227658 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:58.230280 (+ 2622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:58.231015 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:58.231025 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:58.231027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:58.231145 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:58.231562 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:58.231567 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:58.231571 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:58.231828 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:58.231990 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:58.232666 (+ 676us) server_negotiation.cc:299] Negotiation successful
0620 18:35:58.232879 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 18:35:58.234222 26067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:58.215755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:33321 (local address 127.22.37.2:57975)
0620 18:35:58.216425 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:58.216461 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:58.218137 (+ 1676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:58.218749 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:58.218756 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:58.219204 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:58.220005 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:58.220031 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:58.221156 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:58.221161 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:58.222887 (+ 1726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:58.222901 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:58.223667 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:58.223677 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:58.224012 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:58.224762 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:35:58.224800 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:58.227378 (+ 2578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:35:58.230440 (+ 3062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:58.230450 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:58.230468 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:58.230886 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:58.231255 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:58.231261 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:58.231264 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:58.231446 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:58.231967 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:35:58.231974 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:35:58.232303 (+ 329us) client_negotiation.cc:769] Sending connection context
0620 18:35:58.232560 (+ 257us) client_negotiation.cc:240] Negotiation successful
0620 18:35:58.233086 (+ 526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":259,"threads_started":1}
I20260620 18:35:58.235746 26065 heartbeater.cc:347] Connected to a master server at 127.22.37.62:33321
I20260620 18:35:58.236057 26065 heartbeater.cc:464] Registering TS with master...
I20260620 18:35:58.236760 26065 heartbeater.cc:511] Master 127.22.37.62:33321 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.22.37.3@KRBTEST.COM; defaulting to no policy
I20260620 18:35:58.238260 25742 ts_manager.cc:194] Registered new tserver with Master: 3fd51f0f60d847789b072436c34bcfa9 (127.22.37.2:42727)
I20260620 18:35:58.239038 25742 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.37.2@KRBTEST.COM'} at 127.22.37.2:57975
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.37.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.37.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.3.keytab.
Entry for principal kudu/127.22.37.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.3.keytab.
I20260620 18:35:58.262562 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980527431848-22676-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980527431848-22676-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980527431848-22676-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980527431848-22676-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.37.3:0
--local_ip_for_outbound_sockets=127.22.37.3
--webserver_interface=127.22.37.3
--webserver_port=0
--tserver_master_addrs=127.22.37.62:33321
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.3.keytab
--principal=kudu/127.22.37.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:45769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:35:58.574352 26070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:35:58.574680 26070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:35:58.574783 26070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:35:58.582335 26070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:35:58.582461 26070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:35:58.582588 26070 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.37.3
I20260620 18:35:58.592357 26070 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:45769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980527431848-22676-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980527431848-22676-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.3.keytab
--principal=kudu/127.22.37.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.37.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980527431848-22676-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.37.3
--webserver_port=0
--tserver_master_addrs=127.22.37.62:33321
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.37.3
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980527431848-22676-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:35:58.593818 26070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:35:58.595289 26070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:35:58.608629 26075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:58.608568 26076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:58.608595 26078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:35:58.609846 26070 server_base.cc:1034] running on GCE node
I20260620 18:35:58.610438 26070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:35:58.611385 26070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:35:58.612620 26070 hybrid_clock.cc:648] HybridClock initialized: now 1781980558612571 us; error 40 us; skew 500 ppm
I20260620 18:35:58.612967 26070 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:35:58 dist-test-slave-5lgp krb5kdc[25697](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980558, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:35:58.615957 26070 init.cc:377] Logged in from keytab as kudu/127.22.37.3@KRBTEST.COM (short username kudu)
I20260620 18:35:58.617481 26070 webserver.cc:460] Webserver started at http://127.22.37.3:33313/ using document root <none> and password file <none>
I20260620 18:35:58.618161 26070 fs_manager.cc:359] Metadata directory not provided
I20260620 18:35:58.618290 26070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:35:58.618597 26070 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:35:58.621460 26070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980527431848-22676-0/minicluster-data/ts-2/data/instance:
uuid: "f64726d7feb44978b40220dd8c77c858"
format_stamp: "Formatted at 2026-06-20 18:35:58 on dist-test-slave-5lgp"
server_key: "ac484feed05b89b64a3d5fe29df46c5f"
server_key_iv: "9f3c973138fe0852b5581d8dd041600d"
server_key_version: "encryptionkey@0"
I20260620 18:35:58.622234 26070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980527431848-22676-0/minicluster-data/ts-2/wal/instance:
uuid: "f64726d7feb44978b40220dd8c77c858"
format_stamp: "Formatted at 2026-06-20 18:35:58 on dist-test-slave-5lgp"
server_key: "ac484feed05b89b64a3d5fe29df46c5f"
server_key_iv: "9f3c973138fe0852b5581d8dd041600d"
server_key_version: "encryptionkey@0"
I20260620 18:35:58.627099 26070 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:35:58.630427 26085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:58.631640 26070 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:35:58.631794 26070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980527431848-22676-0/minicluster-data/ts-2/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980527431848-22676-0/minicluster-data/ts-2/wal
uuid: "f64726d7feb44978b40220dd8c77c858"
format_stamp: "Formatted at 2026-06-20 18:35:58 on dist-test-slave-5lgp"
server_key: "ac484feed05b89b64a3d5fe29df46c5f"
server_key_iv: "9f3c973138fe0852b5581d8dd041600d"
server_key_version: "encryptionkey@0"
I20260620 18:35:58.631978 26070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980527431848-22676-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980527431848-22676-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980527431848-22676-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:35:58.653227 26070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:35:58.657612 26070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:35:58.657910 26070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:35:58.658849 26070 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:35:58.660111 26070 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:35:58.660225 26070 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:58.660372 26070 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:35:58.660449 26070 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:58.692413 26070 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.3:37469
I20260620 18:35:58.692464 26197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.3:37469 every 8 connection(s)
I20260620 18:35:58.693693 26070 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980527431848-22676-0/minicluster-data/ts-2/data/info.pb
I20260620 18:35:58.699105 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 26070
I20260620 18:35:58.699295 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980527431848-22676-0/minicluster-data/ts-2/wal/instance
I20260620 18:35:58.699883 22676 external_mini_cluster.cc:1428] Setting key 866265c4fa71a39c601775c8b7de4675
Jun 20 18:35:58 dist-test-slave-5lgp krb5kdc[25697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980558, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.3@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM
I20260620 18:35:58.713186 25933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:58.696193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.3:35101 (local address 127.22.37.62:33321)
0620 18:35:58.696374 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:58.696379 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:58.697588 (+ 1209us) server_negotiation.cc:407] Connection header received
0620 18:35:58.699377 (+ 1789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:58.699382 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:58.699478 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:58.699589 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:58.701492 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:58.702594 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:58.704181 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:58.704570 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:58.704703 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:58.707930 (+ 3227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:58.707938 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:58.707946 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:58.707977 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:58.710284 (+ 2307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:58.710976 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:58.710983 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:58.710991 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:58.711107 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:58.711527 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:58.711532 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:58.711535 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:58.711802 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:58.711952 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:58.712619 (+ 667us) server_negotiation.cc:299] Negotiation successful
0620 18:35:58.712828 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 18:35:58.714347 26200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:58.696638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:33321 (local address 127.22.37.3:35101)
0620 18:35:58.697365 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:58.697458 (+ 93us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:58.699108 (+ 1650us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:58.699868 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:58.699879 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:58.700501 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:58.701288 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:58.701319 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:58.702788 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:58.702794 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:58.704034 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:58.704045 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:58.704792 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:58.704800 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:58.705030 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:58.705668 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:35:58.705693 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:58.707735 (+ 2042us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:35:58.710419 (+ 2684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:58.710427 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:58.710442 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:58.710852 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:58.711223 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:58.711228 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:58.711232 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:58.711411 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:58.711928 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:35:58.711937 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:35:58.712261 (+ 324us) client_negotiation.cc:769] Sending connection context
0620 18:35:58.712514 (+ 253us) client_negotiation.cc:240] Negotiation successful
0620 18:35:58.713031 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":297,"threads_started":1}
I20260620 18:35:58.715847 26198 heartbeater.cc:347] Connected to a master server at 127.22.37.62:33321
I20260620 18:35:58.716106 26198 heartbeater.cc:464] Registering TS with master...
I20260620 18:35:58.716763 26198 heartbeater.cc:511] Master 127.22.37.62:33321 requested a full tablet report, sending...
I20260620 18:35:58.718196 25741 ts_manager.cc:194] Registered new tserver with Master: f64726d7feb44978b40220dd8c77c858 (127.22.37.3:37469)
I20260620 18:35:58.719097 25741 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.37.3@KRBTEST.COM'} at 127.22.37.3:35101
I20260620 18:35:58.727878 22676 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:35:58.733906 25931 heartbeater.cc:503] Master 127.22.37.62:33321 was elected leader, sending a full tablet report...
Jun 20 18:35:58 dist-test-slave-5lgp krb5kdc[25697](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980558, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 20 18:35:58 dist-test-slave-5lgp krb5kdc[25697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980558, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM
I20260620 18:35:58.775815 25933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:58.761742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35858 (local address 127.22.37.62:33321)
0620 18:35:58.761940 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:58.761947 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:58.762418 (+ 471us) server_negotiation.cc:407] Connection header received
0620 18:35:58.762627 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:58.762633 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:58.762711 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:58.762816 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:58.764165 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:58.764925 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:58.766344 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:58.766800 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:58.767080 (+ 280us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:58.770907 (+ 3827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:58.770912 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:58.770915 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:58.770949 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:58.773241 (+ 2292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:58.773887 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:58.773893 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:58.773901 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:58.774031 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:58.774413 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:58.774418 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:58.774421 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:58.774738 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:58.774986 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:58.775266 (+ 280us) server_negotiation.cc:299] Negotiation successful
0620 18:35:58.775474 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260620 18:35:58.776355 25741 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:35858
I20260620 18:35:58.778108 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 25803
Jun 20 18:35:58 dist-test-slave-5lgp krb5kdc[25697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980558, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.22.37.1@KRBTEST.COM
I20260620 18:35:58.794278 26209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:58.779098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54876 (local address 127.22.37.1:32991)
0620 18:35:58.779648 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:58.779656 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:58.779686 (+ 30us) server_negotiation.cc:407] Connection header received
0620 18:35:58.779738 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:58.779745 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:58.779954 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:58.780074 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:58.781158 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:58.782209 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:58.783344 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:58.783910 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:58.784002 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:58.787883 (+ 3881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:58.787897 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:58.787915 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:58.787950 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:58.791762 (+ 3812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:58.792440 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:58.792447 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:58.792449 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:58.792564 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:58.792866 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:58.792869 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:58.792871 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:58.793183 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:58.793461 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:58.793690 (+ 229us) server_negotiation.cc:299] Negotiation successful
0620 18:35:58.793877 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":252,"threads_started":1}
W20260620 18:35:58.795557 25926 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:54876
W20260620 18:35:58.796669 22676 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 25803: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:35:58.796852 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 25803
I20260620 18:35:58.812901 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 25937
Jun 20 18:35:58 dist-test-slave-5lgp krb5kdc[25697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980558, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.22.37.2@KRBTEST.COM
I20260620 18:35:58.827601 26210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:58.813794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33814 (local address 127.22.37.2:42727)
0620 18:35:58.814377 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:58.814384 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:58.814410 (+ 26us) server_negotiation.cc:407] Connection header received
0620 18:35:58.814469 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:58.814477 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:58.814689 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:58.814824 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:58.815978 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:58.816956 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:58.818193 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:58.818731 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:58.818825 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:58.822612 (+ 3787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:58.822623 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:58.822638 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:58.822675 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:58.825498 (+ 2823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:58.826018 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:58.826024 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:58.826026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:58.826153 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:58.826432 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:58.826435 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:58.826437 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:58.826726 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:58.826932 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:58.827135 (+ 203us) server_negotiation.cc:299] Negotiation successful
0620 18:35:58.827330 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":288,"threads_started":1}
W20260620 18:35:58.828866 26060 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:33814
W20260620 18:35:58.830376 22676 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 25937: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:35:58.830511 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 25937
I20260620 18:35:58.846067 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 26070
Jun 20 18:35:58 dist-test-slave-5lgp krb5kdc[25697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980558, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.22.37.3@KRBTEST.COM
I20260620 18:35:58.861616 26211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:58.846899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48064 (local address 127.22.37.3:37469)
0620 18:35:58.847559 (+ 660us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:58.847585 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:58.847614 (+ 29us) server_negotiation.cc:407] Connection header received
0620 18:35:58.847676 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:58.847682 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:58.847884 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:58.848015 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:58.849225 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:58.850395 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:58.851526 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:58.852058 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:58.852162 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:58.856336 (+ 4174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:58.856347 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:58.856362 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:58.856395 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:58.859207 (+ 2812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:58.859783 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:58.859790 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:58.859792 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:58.859902 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:58.860191 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:58.860194 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:58.860196 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:58.860512 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:58.860784 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:58.861004 (+ 220us) server_negotiation.cc:299] Negotiation successful
0620 18:35:58.861198 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":332,"threads_started":1}
W20260620 18:35:58.862847 26193 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:48064
W20260620 18:35:58.863911 22676 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 26070: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:35:58.864071 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 26070
I20260620 18:35:58.879451 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 25710
I20260620 18:35:58.880649 25772 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:35:58.982273 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 25710
2026-06-20T18:35:58Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3569 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980527431848-22676-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:35:59 dist-test-slave-5lgp krb5kdc[26215](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:35:59 dist-test-slave-5lgp krb5kdc[26215](info): set up 2 sockets
Jun 20 18:35:59 dist-test-slave-5lgp krb5kdc[26215](info): commencing operation
krb5kdc: starting...
W20260620 18:36:00.061852 22676 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.001s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980527431848-22676-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980527431848-22676-0/krb5kdc/test-user.keytab.
Jun 20 18:36:00 dist-test-slave-5lgp krb5kdc[26215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980560, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:36:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:36:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.37.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.37.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.37.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab.
Entry for principal kudu/127.22.37.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab.
I20260620 18:36:00.248677 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980527431848-22676-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980527431848-22676-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980527431848-22676-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.62:33603
--webserver_interface=127.22.37.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab
--principal=kudu/127.22.37.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:42803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.37.62:33603
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:36:00.563429 26228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:36:00.563803 26228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:36:00.563984 26228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:36:00.571147 26228 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:36:00.571244 26228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:36:00.571290 26228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:36:00.571344 26228 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:36:00.571388 26228 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:36:00.581020 26228 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:42803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980527431848-22676-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980527431848-22676-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.37.62:33603
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab
--principal=kudu/127.22.37.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.62:33603
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.37.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980527431848-22676-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:36:00.582361 26228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:36:00.583647 26228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:36:00.595196 26234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:00.595196 26236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:00.595223 26233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:00.596200 26228 server_base.cc:1034] running on GCE node
I20260620 18:36:00.596930 26228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:36:00.598393 26228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:36:00.599629 26228 hybrid_clock.cc:648] HybridClock initialized: now 1781980560599587 us; error 48 us; skew 500 ppm
I20260620 18:36:00.599965 26228 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:36:00 dist-test-slave-5lgp krb5kdc[26215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980560, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:36:00.603050 26228 init.cc:377] Logged in from keytab as kudu/127.22.37.62@KRBTEST.COM (short username kudu)
I20260620 18:36:00.604643 26228 webserver.cc:460] Webserver started at http://127.22.37.62:36923/ using document root <none> and password file <none>
I20260620 18:36:00.605319 26228 fs_manager.cc:359] Metadata directory not provided
I20260620 18:36:00.605494 26228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:36:00.605810 26228 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:36:00.608706 26228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980527431848-22676-0/minicluster-data/master-0/data/instance:
uuid: "19a36860d2444ef6be8fdaca95216c92"
format_stamp: "Formatted at 2026-06-20 18:36:00 on dist-test-slave-5lgp"
server_key: "ae684f6d6c15a7c4f2fb4338a766627b"
server_key_iv: "8bee0c3aa24d5424a0d64845f966b9dc"
server_key_version: "encryptionkey@0"
I20260620 18:36:00.609565 26228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980527431848-22676-0/minicluster-data/master-0/wal/instance:
uuid: "19a36860d2444ef6be8fdaca95216c92"
format_stamp: "Formatted at 2026-06-20 18:36:00 on dist-test-slave-5lgp"
server_key: "ae684f6d6c15a7c4f2fb4338a766627b"
server_key_iv: "8bee0c3aa24d5424a0d64845f966b9dc"
server_key_version: "encryptionkey@0"
I20260620 18:36:00.614524 26228 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 18:36:00.617841 26243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:00.619119 26228 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 18:36:00.619293 26228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980527431848-22676-0/minicluster-data/master-0/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980527431848-22676-0/minicluster-data/master-0/wal
uuid: "19a36860d2444ef6be8fdaca95216c92"
format_stamp: "Formatted at 2026-06-20 18:36:00 on dist-test-slave-5lgp"
server_key: "ae684f6d6c15a7c4f2fb4338a766627b"
server_key_iv: "8bee0c3aa24d5424a0d64845f966b9dc"
server_key_version: "encryptionkey@0"
I20260620 18:36:00.619485 26228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980527431848-22676-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980527431848-22676-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980527431848-22676-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:36:00.637413 26228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:36:00.641218 26228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:36:00.641525 26228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:36:00.659893 26228 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.62:33603
I20260620 18:36:00.659922 26294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.62:33603 every 8 connection(s)
I20260620 18:36:00.661407 26228 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
I20260620 18:36:00.665066 26295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:36:00.666014 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 26228
I20260620 18:36:00.666242 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980527431848-22676-0/minicluster-data/master-0/wal/instance
I20260620 18:36:00.667060 22676 external_mini_cluster.cc:1428] Setting key 84426547463f8deed8d169128d4c4851
I20260620 18:36:00.677276 26295 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19a36860d2444ef6be8fdaca95216c92: Bootstrap starting.
Jun 20 18:36:00 dist-test-slave-5lgp krb5kdc[26215](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980560, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM
I20260620 18:36:00.680482 26295 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 19a36860d2444ef6be8fdaca95216c92: Neither blocks nor log segments found. Creating new log.
I20260620 18:36:00.681564 26295 log.cc:824] T 00000000000000000000000000000000 P 19a36860d2444ef6be8fdaca95216c92: Log is configured to *not* fsync() on all Append() calls
I20260620 18:36:00.684353 26295 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19a36860d2444ef6be8fdaca95216c92: No bootstrap required, opened a new log
I20260620 18:36:00.688166 26298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:00.669110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33250 (local address 127.22.37.62:33603)
0620 18:36:00.669852 (+ 742us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:00.669867 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:00.669904 (+ 37us) server_negotiation.cc:407] Connection header received
0620 18:36:00.671536 (+ 1632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:00.671562 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:00.671959 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:00.672345 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:00.673789 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:00.675042 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:00.676253 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:00.676908 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:00.677116 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:00.681158 (+ 4042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:00.681165 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:00.681179 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:00.681214 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:00.684307 (+ 3093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:00.684980 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:00.684987 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:00.684993 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:00.685149 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:00.685576 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:00.685582 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:00.685586 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:00.686091 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:00.686429 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:00.686849 (+ 420us) server_negotiation.cc:299] Negotiation successful
0620 18:36:00.687223 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":311,"threads_started":1}
I20260620 18:36:00.692126 26295 raft_consensus.cc:348] T 00000000000000000000000000000000 P 19a36860d2444ef6be8fdaca95216c92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19a36860d2444ef6be8fdaca95216c92" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 33603 } }
I20260620 18:36:00.692505 26295 raft_consensus.cc:374] T 00000000000000000000000000000000 P 19a36860d2444ef6be8fdaca95216c92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:36:00.692608 26295 raft_consensus.cc:729] T 00000000000000000000000000000000 P 19a36860d2444ef6be8fdaca95216c92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19a36860d2444ef6be8fdaca95216c92, State: Initialized, Role: FOLLOWER
I20260620 18:36:00.693123 26295 consensus_queue.cc:260] T 00000000000000000000000000000000 P 19a36860d2444ef6be8fdaca95216c92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19a36860d2444ef6be8fdaca95216c92" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 33603 } }
I20260620 18:36:00.693424 26295 raft_consensus.cc:388] T 00000000000000000000000000000000 P 19a36860d2444ef6be8fdaca95216c92 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:36:00.693560 26295 raft_consensus.cc:482] T 00000000000000000000000000000000 P 19a36860d2444ef6be8fdaca95216c92 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:36:00.693758 26295 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 19a36860d2444ef6be8fdaca95216c92 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:36:00.696017 26295 raft_consensus.cc:504] T 00000000000000000000000000000000 P 19a36860d2444ef6be8fdaca95216c92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19a36860d2444ef6be8fdaca95216c92" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 33603 } }
I20260620 18:36:00.696485 26295 leader_election.cc:302] T 00000000000000000000000000000000 P 19a36860d2444ef6be8fdaca95216c92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 19a36860d2444ef6be8fdaca95216c92; no voters:
I20260620 18:36:00.697203 26295 leader_election.cc:288] T 00000000000000000000000000000000 P 19a36860d2444ef6be8fdaca95216c92 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:36:00.697345 26300 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 19a36860d2444ef6be8fdaca95216c92 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:36:00.698019 26300 raft_consensus.cc:686] T 00000000000000000000000000000000 P 19a36860d2444ef6be8fdaca95216c92 [term 1 LEADER]: Becoming Leader. State: Replica: 19a36860d2444ef6be8fdaca95216c92, State: Running, Role: LEADER
I20260620 18:36:00.698544 26300 consensus_queue.cc:237] T 00000000000000000000000000000000 P 19a36860d2444ef6be8fdaca95216c92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19a36860d2444ef6be8fdaca95216c92" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 33603 } }
I20260620 18:36:00.699487 26295 sys_catalog.cc:564] T 00000000000000000000000000000000 P 19a36860d2444ef6be8fdaca95216c92 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:36:00.701314 26301 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19a36860d2444ef6be8fdaca95216c92 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "19a36860d2444ef6be8fdaca95216c92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19a36860d2444ef6be8fdaca95216c92" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 33603 } } }
I20260620 18:36:00.701735 26301 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19a36860d2444ef6be8fdaca95216c92 [sys.catalog]: This master's current role is: LEADER
I20260620 18:36:00.702057 26302 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19a36860d2444ef6be8fdaca95216c92 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 19a36860d2444ef6be8fdaca95216c92. Latest consensus state: current_term: 1 leader_uuid: "19a36860d2444ef6be8fdaca95216c92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19a36860d2444ef6be8fdaca95216c92" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 33603 } } }
I20260620 18:36:00.702415 26302 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19a36860d2444ef6be8fdaca95216c92 [sys.catalog]: This master's current role is: LEADER
I20260620 18:36:00.706509 26305 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:36:00.712296 26305 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:36:00.720840 26305 catalog_manager.cc:1347] Generated new cluster ID: 7e573bb05d4b405c90819d2917b47fe2
I20260620 18:36:00.720980 26305 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:36:00.738595 26305 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:36:00.740540 26305 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:36:00.754119 26305 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 19a36860d2444ef6be8fdaca95216c92: Generated new TSK 0
I20260620 18:36:00.755086 26305 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.37.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.37.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.37.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.1.keytab.
Entry for principal kudu/127.22.37.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.1.keytab.
I20260620 18:36:00.811657 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980527431848-22676-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980527431848-22676-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980527431848-22676-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980527431848-22676-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.37.1:0
--local_ip_for_outbound_sockets=127.22.37.1
--webserver_interface=127.22.37.1
--webserver_port=0
--tserver_master_addrs=127.22.37.62:33603
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.1.keytab
--principal=kudu/127.22.37.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:42803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:36:01.139753 26321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:36:01.140141 26321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:36:01.140317 26321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:36:01.148190 26321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:36:01.148291 26321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:36:01.148478 26321 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.37.1
I20260620 18:36:01.158121 26321 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:42803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980527431848-22676-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980527431848-22676-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.1.keytab
--principal=kudu/127.22.37.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.37.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980527431848-22676-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.37.1
--webserver_port=0
--tserver_master_addrs=127.22.37.62:33603
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.37.1
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980527431848-22676-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:36:01.159440 26321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:36:01.160756 26321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:36:01.174077 26327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:01.174108 26329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:01.174672 26326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:01.174947 26321 server_base.cc:1034] running on GCE node
I20260620 18:36:01.175534 26321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:36:01.176388 26321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:36:01.177577 26321 hybrid_clock.cc:648] HybridClock initialized: now 1781980561177531 us; error 45 us; skew 500 ppm
I20260620 18:36:01.177852 26321 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:36:01 dist-test-slave-5lgp krb5kdc[26215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980561, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:36:01.180848 26321 init.cc:377] Logged in from keytab as kudu/127.22.37.1@KRBTEST.COM (short username kudu)
I20260620 18:36:01.182310 26321 webserver.cc:460] Webserver started at http://127.22.37.1:35327/ using document root <none> and password file <none>
I20260620 18:36:01.182943 26321 fs_manager.cc:359] Metadata directory not provided
I20260620 18:36:01.183100 26321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:36:01.183450 26321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:36:01.186298 26321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980527431848-22676-0/minicluster-data/ts-0/data/instance:
uuid: "83ec136f8cbd445d8a85ff58019cd147"
format_stamp: "Formatted at 2026-06-20 18:36:01 on dist-test-slave-5lgp"
server_key: "1a86ebd30317a85ed82d9ed72684f3ed"
server_key_iv: "7c523dce2d57450103bf0e544f05a6b9"
server_key_version: "encryptionkey@0"
I20260620 18:36:01.186977 26321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980527431848-22676-0/minicluster-data/ts-0/wal/instance:
uuid: "83ec136f8cbd445d8a85ff58019cd147"
format_stamp: "Formatted at 2026-06-20 18:36:01 on dist-test-slave-5lgp"
server_key: "1a86ebd30317a85ed82d9ed72684f3ed"
server_key_iv: "7c523dce2d57450103bf0e544f05a6b9"
server_key_version: "encryptionkey@0"
I20260620 18:36:01.191722 26321 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 18:36:01.195132 26336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:01.196449 26321 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 18:36:01.196573 26321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980527431848-22676-0/minicluster-data/ts-0/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980527431848-22676-0/minicluster-data/ts-0/wal
uuid: "83ec136f8cbd445d8a85ff58019cd147"
format_stamp: "Formatted at 2026-06-20 18:36:01 on dist-test-slave-5lgp"
server_key: "1a86ebd30317a85ed82d9ed72684f3ed"
server_key_iv: "7c523dce2d57450103bf0e544f05a6b9"
server_key_version: "encryptionkey@0"
I20260620 18:36:01.196735 26321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980527431848-22676-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980527431848-22676-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980527431848-22676-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:36:01.218240 26321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:36:01.222647 26321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:36:01.222930 26321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:36:01.223953 26321 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:36:01.225240 26321 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:36:01.225351 26321 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:01.225517 26321 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:36:01.225586 26321 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:01.258584 26321 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.1:39665
I20260620 18:36:01.258670 26448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.1:39665 every 8 connection(s)
I20260620 18:36:01.259865 26321 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980527431848-22676-0/minicluster-data/ts-0/data/info.pb
I20260620 18:36:01.263006 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 26321
I20260620 18:36:01.263273 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980527431848-22676-0/minicluster-data/ts-0/wal/instance
I20260620 18:36:01.263989 22676 external_mini_cluster.cc:1428] Setting key 30acc1f9293d8274f207b4fd0caed9c7
Jun 20 18:36:01 dist-test-slave-5lgp krb5kdc[26215](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980561, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.1@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM
I20260620 18:36:01.284641 26451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:01.262483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.1:46985 (local address 127.22.37.62:33603)
0620 18:36:01.263015 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:01.263023 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:01.264147 (+ 1124us) server_negotiation.cc:407] Connection header received
0620 18:36:01.266151 (+ 2004us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:01.266158 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:01.266251 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:01.266376 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:01.267759 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:01.268953 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:01.270479 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:01.271284 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:01.271447 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:01.276667 (+ 5220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:01.276677 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:01.276685 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:01.276730 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:01.280563 (+ 3833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:01.281597 (+ 1034us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:01.281603 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:01.281607 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:01.281757 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:01.282273 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:01.282278 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:01.282284 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:01.282767 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:01.282955 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:01.283977 (+ 1022us) server_negotiation.cc:299] Negotiation successful
0620 18:36:01.284214 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":249,"threads_started":1}
WARNING: no policy specified for kudu/127.22.37.2@KRBTEST.COM; defaulting to no policy
I20260620 18:36:01.285738 26452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:01.263065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:33603 (local address 127.22.37.1:46985)
0620 18:36:01.263916 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:01.263966 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:01.265861 (+ 1895us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:01.266552 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:01.266559 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:01.267057 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:01.267578 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:01.267599 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:01.269093 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:01.269096 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:01.270333 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:01.270344 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:01.271782 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:01.271790 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:01.272080 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:01.273016 (+ 936us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:01.273044 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:01.276375 (+ 3331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:36:01.280715 (+ 4340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:01.280722 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:01.280737 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:01.281449 (+ 712us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:01.281883 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:01.281890 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:01.281894 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:01.282103 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:01.282927 (+ 824us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:36:01.282936 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:36:01.283508 (+ 572us) client_negotiation.cc:769] Sending connection context
0620 18:36:01.283859 (+ 351us) client_negotiation.cc:240] Negotiation successful
0620 18:36:01.284438 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":336,"threads_started":1}
I20260620 18:36:01.287393 26449 heartbeater.cc:347] Connected to a master server at 127.22.37.62:33603
I20260620 18:36:01.287708 26449 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.37.2@KRBTEST.COM" created.
I20260620 18:36:01.288509 26449 heartbeater.cc:511] Master 127.22.37.62:33603 requested a full tablet report, sending...
I20260620 18:36:01.290640 26260 ts_manager.cc:194] Registered new tserver with Master: 83ec136f8cbd445d8a85ff58019cd147 (127.22.37.1:39665)
I20260620 18:36:01.292054 26260 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.37.1@KRBTEST.COM'} at 127.22.37.1:46985
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.37.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.2.keytab.
Entry for principal kudu/127.22.37.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.2.keytab.
I20260620 18:36:01.310827 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980527431848-22676-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980527431848-22676-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980527431848-22676-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980527431848-22676-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.37.2:0
--local_ip_for_outbound_sockets=127.22.37.2
--webserver_interface=127.22.37.2
--webserver_port=0
--tserver_master_addrs=127.22.37.62:33603
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.2.keytab
--principal=kudu/127.22.37.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:42803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:36:01.624321 26455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:36:01.624619 26455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:36:01.624713 26455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:36:01.632098 26455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:36:01.632226 26455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:36:01.632349 26455 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.37.2
I20260620 18:36:01.642191 26455 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:42803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980527431848-22676-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980527431848-22676-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.2.keytab
--principal=kudu/127.22.37.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.37.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980527431848-22676-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.37.2
--webserver_port=0
--tserver_master_addrs=127.22.37.62:33603
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.37.2
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980527431848-22676-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:36:01.643653 26455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:36:01.644945 26455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:36:01.657955 26463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:01.658046 26460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:01.658027 26461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:01.658995 26455 server_base.cc:1034] running on GCE node
I20260620 18:36:01.659610 26455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:36:01.660627 26455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:36:01.661852 26455 hybrid_clock.cc:648] HybridClock initialized: now 1781980561661781 us; error 57 us; skew 500 ppm
I20260620 18:36:01.662191 26455 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:36:01 dist-test-slave-5lgp krb5kdc[26215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980561, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:36:01.665691 26455 init.cc:377] Logged in from keytab as kudu/127.22.37.2@KRBTEST.COM (short username kudu)
I20260620 18:36:01.667308 26455 webserver.cc:460] Webserver started at http://127.22.37.2:33009/ using document root <none> and password file <none>
I20260620 18:36:01.668035 26455 fs_manager.cc:359] Metadata directory not provided
I20260620 18:36:01.668179 26455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:36:01.668490 26455 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:36:01.671464 26455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980527431848-22676-0/minicluster-data/ts-1/data/instance:
uuid: "d04880be14ab4ac9934b101e254ba46a"
format_stamp: "Formatted at 2026-06-20 18:36:01 on dist-test-slave-5lgp"
server_key: "b773b60f814d615650d4c428041a9363"
server_key_iv: "5d360a4082d84134878a5ebd1830c2d6"
server_key_version: "encryptionkey@0"
I20260620 18:36:01.672238 26455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980527431848-22676-0/minicluster-data/ts-1/wal/instance:
uuid: "d04880be14ab4ac9934b101e254ba46a"
format_stamp: "Formatted at 2026-06-20 18:36:01 on dist-test-slave-5lgp"
server_key: "b773b60f814d615650d4c428041a9363"
server_key_iv: "5d360a4082d84134878a5ebd1830c2d6"
server_key_version: "encryptionkey@0"
I20260620 18:36:01.677460 26455 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260620 18:36:01.680838 26470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:01.682191 26455 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:36:01.682359 26455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980527431848-22676-0/minicluster-data/ts-1/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980527431848-22676-0/minicluster-data/ts-1/wal
uuid: "d04880be14ab4ac9934b101e254ba46a"
format_stamp: "Formatted at 2026-06-20 18:36:01 on dist-test-slave-5lgp"
server_key: "b773b60f814d615650d4c428041a9363"
server_key_iv: "5d360a4082d84134878a5ebd1830c2d6"
server_key_version: "encryptionkey@0"
I20260620 18:36:01.682564 26455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980527431848-22676-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980527431848-22676-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980527431848-22676-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:36:01.706990 26455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:36:01.711199 26455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:36:01.711489 26455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:36:01.712450 26455 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:36:01.713804 26455 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:36:01.713903 26455 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:01.714048 26455 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:36:01.714125 26455 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:01.747079 26455 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.2:42713
I20260620 18:36:01.747128 26582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.2:42713 every 8 connection(s)
I20260620 18:36:01.748435 26455 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980527431848-22676-0/minicluster-data/ts-1/data/info.pb
I20260620 18:36:01.758157 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 26455
I20260620 18:36:01.758347 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980527431848-22676-0/minicluster-data/ts-1/wal/instance
I20260620 18:36:01.758772 22676 external_mini_cluster.cc:1428] Setting key 9d599c25ab674b7c7afeee022e30b949
Jun 20 18:36:01 dist-test-slave-5lgp krb5kdc[26215](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980561, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.2@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM
I20260620 18:36:01.768826 26451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:01.750963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.2:56509 (local address 127.22.37.62:33603)
0620 18:36:01.751134 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:01.751139 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:01.752290 (+ 1151us) server_negotiation.cc:407] Connection header received
0620 18:36:01.754155 (+ 1865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:01.754159 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:01.754253 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:01.754365 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:01.755727 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:01.756422 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:01.758019 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:01.758598 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:01.758734 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:01.762781 (+ 4047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:01.762786 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:01.762789 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:01.762820 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:01.765340 (+ 2520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:01.766323 (+ 983us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:01.766330 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:01.766336 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:01.766518 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:01.766906 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:01.766913 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:01.766919 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:01.767305 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:01.767463 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:01.768108 (+ 645us) server_negotiation.cc:299] Negotiation successful
0620 18:36:01.768382 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 18:36:01.769651 26585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:01.751458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:33603 (local address 127.22.37.2:56509)
0620 18:36:01.752138 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:01.752175 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:01.753858 (+ 1683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:01.754549 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:01.754557 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:01.755043 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:01.755589 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:01.755606 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:01.756557 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:01.756561 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:01.757874 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:01.757885 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:01.758828 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:01.758836 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:01.759059 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:01.759745 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:01.759775 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:01.762515 (+ 2740us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:36:01.765566 (+ 3051us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:01.765579 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:01.765600 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:01.766197 (+ 597us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:01.766625 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:01.766628 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:01.766631 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:01.766795 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:01.767436 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:36:01.767441 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:36:01.767739 (+ 298us) client_negotiation.cc:769] Sending connection context
0620 18:36:01.767962 (+ 223us) client_negotiation.cc:240] Negotiation successful
0620 18:36:01.768335 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":271,"threads_started":1}
I20260620 18:36:01.771406 26583 heartbeater.cc:347] Connected to a master server at 127.22.37.62:33603
I20260620 18:36:01.771723 26583 heartbeater.cc:464] Registering TS with master...
I20260620 18:36:01.772433 26583 heartbeater.cc:511] Master 127.22.37.62:33603 requested a full tablet report, sending...
I20260620 18:36:01.773984 26260 ts_manager.cc:194] Registered new tserver with Master: d04880be14ab4ac9934b101e254ba46a (127.22.37.2:42713)
I20260620 18:36:01.775282 26260 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.37.2@KRBTEST.COM'} at 127.22.37.2:56509
WARNING: no policy specified for kudu/127.22.37.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.37.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.37.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.3.keytab.
Entry for principal kudu/127.22.37.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.3.keytab.
I20260620 18:36:01.802935 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980527431848-22676-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980527431848-22676-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980527431848-22676-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980527431848-22676-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.37.3:0
--local_ip_for_outbound_sockets=127.22.37.3
--webserver_interface=127.22.37.3
--webserver_port=0
--tserver_master_addrs=127.22.37.62:33603
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.3.keytab
--principal=kudu/127.22.37.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:42803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:36:02.127996 26588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:36:02.128396 26588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:36:02.128584 26588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:36:02.136041 26588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:36:02.136215 26588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:36:02.136497 26588 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.37.3
I20260620 18:36:02.153194 26588 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:42803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980527431848-22676-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980527431848-22676-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.3.keytab
--principal=kudu/127.22.37.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.37.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980527431848-22676-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.37.3
--webserver_port=0
--tserver_master_addrs=127.22.37.62:33603
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.37.3
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980527431848-22676-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:36:02.154892 26588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:36:02.156294 26588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:36:02.172307 26594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:02.172341 26593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:02.172869 26596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:02.173273 26588 server_base.cc:1034] running on GCE node
I20260620 18:36:02.173890 26588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:36:02.174809 26588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:36:02.176028 26588 hybrid_clock.cc:648] HybridClock initialized: now 1781980562175965 us; error 59 us; skew 500 ppm
I20260620 18:36:02.176330 26588 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:36:02 dist-test-slave-5lgp krb5kdc[26215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980562, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:36:02.179378 26588 init.cc:377] Logged in from keytab as kudu/127.22.37.3@KRBTEST.COM (short username kudu)
I20260620 18:36:02.180702 26588 webserver.cc:460] Webserver started at http://127.22.37.3:38965/ using document root <none> and password file <none>
I20260620 18:36:02.181349 26588 fs_manager.cc:359] Metadata directory not provided
I20260620 18:36:02.181548 26588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:36:02.181877 26588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:36:02.184603 26588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980527431848-22676-0/minicluster-data/ts-2/data/instance:
uuid: "c31b95f33251478e8ec88fb5049b9f13"
format_stamp: "Formatted at 2026-06-20 18:36:02 on dist-test-slave-5lgp"
server_key: "a190f6ee0aeb9a0d86f23eb8be7acff0"
server_key_iv: "c9779f5f62237a2bf954578aa1ed527b"
server_key_version: "encryptionkey@0"
I20260620 18:36:02.185297 26588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980527431848-22676-0/minicluster-data/ts-2/wal/instance:
uuid: "c31b95f33251478e8ec88fb5049b9f13"
format_stamp: "Formatted at 2026-06-20 18:36:02 on dist-test-slave-5lgp"
server_key: "a190f6ee0aeb9a0d86f23eb8be7acff0"
server_key_iv: "c9779f5f62237a2bf954578aa1ed527b"
server_key_version: "encryptionkey@0"
I20260620 18:36:02.190536 26588 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 18:36:02.194098 26603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:02.195374 26588 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:36:02.195547 26588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980527431848-22676-0/minicluster-data/ts-2/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980527431848-22676-0/minicluster-data/ts-2/wal
uuid: "c31b95f33251478e8ec88fb5049b9f13"
format_stamp: "Formatted at 2026-06-20 18:36:02 on dist-test-slave-5lgp"
server_key: "a190f6ee0aeb9a0d86f23eb8be7acff0"
server_key_iv: "c9779f5f62237a2bf954578aa1ed527b"
server_key_version: "encryptionkey@0"
I20260620 18:36:02.195747 26588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980527431848-22676-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980527431848-22676-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980527431848-22676-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:36:02.214555 26588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:36:02.218302 26588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:36:02.218580 26588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:36:02.219513 26588 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:36:02.220775 26588 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:36:02.220875 26588 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:02.221041 26588 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:36:02.221114 26588 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:02.253162 26588 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.3:38513
I20260620 18:36:02.253196 26715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.3:38513 every 8 connection(s)
I20260620 18:36:02.254501 26588 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980527431848-22676-0/minicluster-data/ts-2/data/info.pb
I20260620 18:36:02.262107 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 26588
I20260620 18:36:02.262293 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980527431848-22676-0/minicluster-data/ts-2/wal/instance
I20260620 18:36:02.262822 22676 external_mini_cluster.cc:1428] Setting key 8bbadcc420c1b027acd814929450e5da
Jun 20 18:36:02 dist-test-slave-5lgp krb5kdc[26215](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980562, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.3@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM
I20260620 18:36:02.273459 26451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:02.257017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.3:41901 (local address 127.22.37.62:33603)
0620 18:36:02.257205 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:02.257209 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:02.258473 (+ 1264us) server_negotiation.cc:407] Connection header received
0620 18:36:02.260214 (+ 1741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:02.260220 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:02.260313 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:02.260424 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:02.261953 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:02.262826 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:02.264581 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:02.265192 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:02.265321 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:02.268589 (+ 3268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:02.268593 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:02.268597 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:02.268625 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:02.271027 (+ 2402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:02.271659 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:02.271663 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:02.271665 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:02.271776 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:02.272110 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:02.272113 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:02.272115 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:02.272340 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:02.272440 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:02.273023 (+ 583us) server_negotiation.cc:299] Negotiation successful
0620 18:36:02.273166 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 18:36:02.274410 26718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:02.257512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:33603 (local address 127.22.37.3:41901)
0620 18:36:02.258277 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:02.258322 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:02.259953 (+ 1631us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:02.260652 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:02.260660 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:02.261158 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:02.261771 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:02.261797 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:02.263026 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:02.263030 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:02.264441 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:02.264452 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:02.265447 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:02.265454 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:02.265729 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:02.266303 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:02.266327 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:02.268380 (+ 2053us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:36:02.271135 (+ 2755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:02.271141 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:02.271153 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:02.271554 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:02.271861 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:02.271864 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:02.271866 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:02.272020 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:02.272434 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:36:02.272450 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:36:02.272726 (+ 276us) client_negotiation.cc:769] Sending connection context
0620 18:36:02.272934 (+ 208us) client_negotiation.cc:240] Negotiation successful
0620 18:36:02.273264 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":293,"threads_started":1}
I20260620 18:36:02.275766 26716 heartbeater.cc:347] Connected to a master server at 127.22.37.62:33603
I20260620 18:36:02.276063 26716 heartbeater.cc:464] Registering TS with master...
I20260620 18:36:02.276865 26716 heartbeater.cc:511] Master 127.22.37.62:33603 requested a full tablet report, sending...
I20260620 18:36:02.278316 26260 ts_manager.cc:194] Registered new tserver with Master: c31b95f33251478e8ec88fb5049b9f13 (127.22.37.3:38513)
I20260620 18:36:02.279049 26260 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.37.3@KRBTEST.COM'} at 127.22.37.3:41901
I20260620 18:36:02.290112 22676 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:36:02.295697 26449 heartbeater.cc:503] Master 127.22.37.62:33603 was elected leader, sending a full tablet report...
Jun 20 18:36:02 dist-test-slave-5lgp krb5kdc[26215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980562, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 18:36:02 dist-test-slave-5lgp krb5kdc[26215](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980562, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.37.1@KRBTEST.COM
I20260620 18:36:02.337102 26723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:02.322625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57706 (local address 127.22.37.1:39665)
0620 18:36:02.323456 (+ 831us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:02.323465 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:02.323503 (+ 38us) server_negotiation.cc:407] Connection header received
0620 18:36:02.323566 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:02.323573 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:02.323801 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:02.323961 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:02.325250 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:02.326320 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:02.327500 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:02.328059 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:02.328154 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:02.332084 (+ 3930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:02.332095 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:02.332111 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:02.332141 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:02.334868 (+ 2727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:02.335442 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:02.335449 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:02.335451 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:02.335562 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:02.335851 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:02.335855 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:02.335856 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:02.336182 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:02.336392 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:02.336627 (+ 235us) server_negotiation.cc:299] Negotiation successful
0620 18:36:02.336812 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":634,"thread_start_us":430,"threads_started":1}
Jun 20 18:36:02 dist-test-slave-5lgp krb5kdc[26215](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980562, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM
I20260620 18:36:02.353642 26451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:02.341355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33262 (local address 127.22.37.62:33603)
0620 18:36:02.341575 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:02.341582 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:02.341723 (+ 141us) server_negotiation.cc:407] Connection header received
0620 18:36:02.341830 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:02.341834 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:02.341902 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:02.341979 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:02.343143 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:02.343816 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:02.344963 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:02.345347 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:02.345498 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:02.349187 (+ 3689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:02.349192 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:02.349195 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:02.349227 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:02.351501 (+ 2274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:02.352013 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:02.352017 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:02.352020 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:02.352132 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:02.352430 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:02.352433 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:02.352434 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:02.352724 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:02.352906 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:02.353141 (+ 235us) server_negotiation.cc:299] Negotiation successful
0620 18:36:02.353289 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
W20260620 18:36:02.354224 26260 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33262
I20260620 18:36:02.355178 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 26321
I20260620 18:36:02.367686 26723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:02.356128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57708 (local address 127.22.37.1:39665)
0620 18:36:02.356330 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:02.356336 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:02.356501 (+ 165us) server_negotiation.cc:407] Connection header received
0620 18:36:02.356676 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:02.356682 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:02.356779 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:02.356902 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:02.358109 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:02.358793 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:02.359987 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:02.360415 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:02.360554 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:02.362779 (+ 2225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:02.362791 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:02.362794 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:02.362826 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:02.365163 (+ 2337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:02.365854 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:02.365860 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:02.365864 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:02.365998 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:02.366363 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:02.366368 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:02.366371 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:02.366622 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:02.366854 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:02.367169 (+ 315us) server_negotiation.cc:299] Negotiation successful
0620 18:36:02.367386 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 18:36:02.368304 26444 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:36:02.449345 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 26321
I20260620 18:36:02.465235 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 26455
Jun 20 18:36:02 dist-test-slave-5lgp krb5kdc[26215](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980562, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.37.2@KRBTEST.COM
I20260620 18:36:02.480219 26728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:02.466110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34946 (local address 127.22.37.2:42713)
0620 18:36:02.466626 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:02.466633 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:02.466659 (+ 26us) server_negotiation.cc:407] Connection header received
0620 18:36:02.466719 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:02.466725 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:02.466937 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:02.467075 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:02.468163 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:02.469178 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:02.470442 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:02.470983 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:02.471076 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:02.475040 (+ 3964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:02.475051 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:02.475067 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:02.475100 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:02.477753 (+ 2653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:02.478351 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:02.478358 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:02.478360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:02.478472 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:02.478760 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:02.478763 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:02.478765 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:02.479073 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:02.479283 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:02.479530 (+ 247us) server_negotiation.cc:299] Negotiation successful
0620 18:36:02.479772 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":273,"threads_started":1}
I20260620 18:36:02.486579 26578 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:36:02.567852 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 26455
I20260620 18:36:02.583559 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 26588
Jun 20 18:36:02 dist-test-slave-5lgp krb5kdc[26215](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980562, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.37.3@KRBTEST.COM
I20260620 18:36:02.599210 26730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:02.584552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46098 (local address 127.22.37.3:38513)
0620 18:36:02.585151 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:02.585158 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:02.585185 (+ 27us) server_negotiation.cc:407] Connection header received
0620 18:36:02.585243 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:02.585250 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:02.585522 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:02.585651 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:02.586957 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:02.587958 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:02.589182 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:02.589878 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:02.589983 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:02.594049 (+ 4066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:02.594060 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:02.594074 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:02.594103 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:02.596863 (+ 2760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:02.597461 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:02.597468 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:02.597471 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:02.597583 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:02.597866 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:02.597869 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:02.597871 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:02.598180 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:02.598382 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:02.598645 (+ 263us) server_negotiation.cc:299] Negotiation successful
0620 18:36:02.598883 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":302,"threads_started":1}
I20260620 18:36:02.600557 26711 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:36:02.687569 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 26588
I20260620 18:36:02.704272 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 26228
I20260620 18:36:02.705555 26290 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:36:02.808175 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 26228
2026-06-20T18:36:02Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3826 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980527431848-22676-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:36:02 dist-test-slave-5lgp krb5kdc[26735](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:36:02 dist-test-slave-5lgp krb5kdc[26735](info): set up 2 sockets
Jun 20 18:36:02 dist-test-slave-5lgp krb5kdc[26735](info): commencing operation
krb5kdc: starting...
W20260620 18:36:03.905821 22676 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.003s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980527431848-22676-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980527431848-22676-0/krb5kdc/test-user.keytab.
Jun 20 18:36:04 dist-test-slave-5lgp krb5kdc[26735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980564, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:36:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:36:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.37.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.37.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.37.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab.
Entry for principal kudu/127.22.37.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab.
I20260620 18:36:04.105865 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980527431848-22676-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980527431848-22676-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980527431848-22676-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.62:32975
--webserver_interface=127.22.37.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab
--principal=kudu/127.22.37.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:33595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.37.62:32975
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:36:04.427313 26748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:36:04.427727 26748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:36:04.427899 26748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:36:04.435452 26748 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:36:04.435633 26748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:36:04.435748 26748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:36:04.435845 26748 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:36:04.435935 26748 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:36:04.446457 26748 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:33595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980527431848-22676-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980527431848-22676-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.37.62:32975
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab
--principal=kudu/127.22.37.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.62:32975
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.37.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980527431848-22676-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:36:04.447911 26748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:36:04.449270 26748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:36:04.461113 26754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:04.461205 26753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:04.461205 26756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:04.462716 26748 server_base.cc:1034] running on GCE node
I20260620 18:36:04.463613 26748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:36:04.465117 26748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:36:04.466315 26748 hybrid_clock.cc:648] HybridClock initialized: now 1781980564466258 us; error 57 us; skew 500 ppm
I20260620 18:36:04.466696 26748 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:36:04 dist-test-slave-5lgp krb5kdc[26735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980564, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:36:04.469885 26748 init.cc:377] Logged in from keytab as kudu/127.22.37.62@KRBTEST.COM (short username kudu)
I20260620 18:36:04.471161 26748 fs_manager.cc:359] Metadata directory not provided
I20260620 18:36:04.471283 26748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:36:04.471650 26748 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:36:04.474875 26748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980527431848-22676-0/minicluster-data/master-0/data/instance:
uuid: "dd43eb11ef5f4047bc4d9ef81617990f"
format_stamp: "Formatted at 2026-06-20 18:36:04 on dist-test-slave-5lgp"
server_key: "e24cf7aeb4ce98067075bb1c97012b73"
server_key_iv: "ef6f8a84ad15e3de91154c41268df1bb"
server_key_version: "encryptionkey@0"
I20260620 18:36:04.475757 26748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980527431848-22676-0/minicluster-data/master-0/wal/instance:
uuid: "dd43eb11ef5f4047bc4d9ef81617990f"
format_stamp: "Formatted at 2026-06-20 18:36:04 on dist-test-slave-5lgp"
server_key: "e24cf7aeb4ce98067075bb1c97012b73"
server_key_iv: "ef6f8a84ad15e3de91154c41268df1bb"
server_key_version: "encryptionkey@0"
I20260620 18:36:04.481097 26748 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260620 18:36:04.484854 26762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:04.486387 26748 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:36:04.486577 26748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980527431848-22676-0/minicluster-data/master-0/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980527431848-22676-0/minicluster-data/master-0/wal
uuid: "dd43eb11ef5f4047bc4d9ef81617990f"
format_stamp: "Formatted at 2026-06-20 18:36:04 on dist-test-slave-5lgp"
server_key: "e24cf7aeb4ce98067075bb1c97012b73"
server_key_iv: "ef6f8a84ad15e3de91154c41268df1bb"
server_key_version: "encryptionkey@0"
I20260620 18:36:04.486828 26748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980527431848-22676-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980527431848-22676-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980527431848-22676-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:36:04.500072 26748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:36:04.504732 26748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:36:04.505044 26748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:36:04.523856 26748 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.62:32975
I20260620 18:36:04.523837 26813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.62:32975 every 8 connection(s)
I20260620 18:36:04.524886 26748 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
I20260620 18:36:04.528692 26814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:36:04.534861 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 26748
I20260620 18:36:04.535151 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980527431848-22676-0/minicluster-data/master-0/wal/instance
I20260620 18:36:04.535785 22676 external_mini_cluster.cc:1428] Setting key c866dd849ee4b22c5a5f9136bd2b0159
I20260620 18:36:04.540426 26814 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd43eb11ef5f4047bc4d9ef81617990f: Bootstrap starting.
I20260620 18:36:04.544550 26814 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dd43eb11ef5f4047bc4d9ef81617990f: Neither blocks nor log segments found. Creating new log.
I20260620 18:36:04.545926 26814 log.cc:824] T 00000000000000000000000000000000 P dd43eb11ef5f4047bc4d9ef81617990f: Log is configured to *not* fsync() on all Append() calls
I20260620 18:36:04.549290 26814 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd43eb11ef5f4047bc4d9ef81617990f: No bootstrap required, opened a new log
Jun 20 18:36:04 dist-test-slave-5lgp krb5kdc[26735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980564, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM
I20260620 18:36:04.559221 26814 raft_consensus.cc:348] T 00000000000000000000000000000000 P dd43eb11ef5f4047bc4d9ef81617990f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd43eb11ef5f4047bc4d9ef81617990f" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 32975 } }
I20260620 18:36:04.559734 26814 raft_consensus.cc:374] T 00000000000000000000000000000000 P dd43eb11ef5f4047bc4d9ef81617990f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:36:04.559855 26814 raft_consensus.cc:729] T 00000000000000000000000000000000 P dd43eb11ef5f4047bc4d9ef81617990f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd43eb11ef5f4047bc4d9ef81617990f, State: Initialized, Role: FOLLOWER
I20260620 18:36:04.560554 26814 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd43eb11ef5f4047bc4d9ef81617990f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd43eb11ef5f4047bc4d9ef81617990f" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 32975 } }
I20260620 18:36:04.560842 26814 raft_consensus.cc:388] T 00000000000000000000000000000000 P dd43eb11ef5f4047bc4d9ef81617990f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:36:04.561007 26814 raft_consensus.cc:482] T 00000000000000000000000000000000 P dd43eb11ef5f4047bc4d9ef81617990f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:36:04.561216 26814 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dd43eb11ef5f4047bc4d9ef81617990f [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:36:04.561939 26817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:04.538001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48080 (local address 127.22.37.62:32975)
0620 18:36:04.538828 (+ 827us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:04.538847 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:04.538901 (+ 54us) server_negotiation.cc:407] Connection header received
0620 18:36:04.541405 (+ 2504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:04.541436 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:04.541970 (+ 534us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:04.542502 (+ 532us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:04.543904 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:04.546026 (+ 2122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:04.547769 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:04.548821 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:04.549018 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:04.552864 (+ 3846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:04.552876 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:04.552897 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:04.552942 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:04.557005 (+ 4063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:04.557836 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:04.557846 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:04.557856 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:04.558071 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:04.558448 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:04.558452 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:04.558455 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:04.559226 (+ 771us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:04.559535 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:04.559888 (+ 353us) server_negotiation.cc:299] Negotiation successful
0620 18:36:04.560346 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":339,"threads_started":1}
I20260620 18:36:04.563938 26814 raft_consensus.cc:504] T 00000000000000000000000000000000 P dd43eb11ef5f4047bc4d9ef81617990f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd43eb11ef5f4047bc4d9ef81617990f" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 32975 } }
I20260620 18:36:04.564517 26814 leader_election.cc:302] T 00000000000000000000000000000000 P dd43eb11ef5f4047bc4d9ef81617990f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd43eb11ef5f4047bc4d9ef81617990f; no voters:
I20260620 18:36:04.565178 26814 leader_election.cc:288] T 00000000000000000000000000000000 P dd43eb11ef5f4047bc4d9ef81617990f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:36:04.565296 26819 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dd43eb11ef5f4047bc4d9ef81617990f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:36:04.565963 26819 raft_consensus.cc:686] T 00000000000000000000000000000000 P dd43eb11ef5f4047bc4d9ef81617990f [term 1 LEADER]: Becoming Leader. State: Replica: dd43eb11ef5f4047bc4d9ef81617990f, State: Running, Role: LEADER
I20260620 18:36:04.566530 26819 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd43eb11ef5f4047bc4d9ef81617990f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd43eb11ef5f4047bc4d9ef81617990f" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 32975 } }
I20260620 18:36:04.567627 26814 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd43eb11ef5f4047bc4d9ef81617990f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:36:04.570715 26821 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd43eb11ef5f4047bc4d9ef81617990f [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd43eb11ef5f4047bc4d9ef81617990f. Latest consensus state: current_term: 1 leader_uuid: "dd43eb11ef5f4047bc4d9ef81617990f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd43eb11ef5f4047bc4d9ef81617990f" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 32975 } } }
I20260620 18:36:04.571098 26821 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd43eb11ef5f4047bc4d9ef81617990f [sys.catalog]: This master's current role is: LEADER
I20260620 18:36:04.571599 26820 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd43eb11ef5f4047bc4d9ef81617990f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dd43eb11ef5f4047bc4d9ef81617990f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd43eb11ef5f4047bc4d9ef81617990f" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 32975 } } }
I20260620 18:36:04.571970 26820 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd43eb11ef5f4047bc4d9ef81617990f [sys.catalog]: This master's current role is: LEADER
I20260620 18:36:04.576834 26828 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:36:04.581969 26828 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:36:04.591879 26828 catalog_manager.cc:1347] Generated new cluster ID: 469e79660ae741e4ba46c31bafc3d8ed
I20260620 18:36:04.591996 26828 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:36:04.634411 26828 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:36:04.636402 26828 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:36:04.647508 26828 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dd43eb11ef5f4047bc4d9ef81617990f: Generated new TSK 0
I20260620 18:36:04.648641 26828 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.37.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.37.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.37.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.1.keytab.
Entry for principal kudu/127.22.37.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.1.keytab.
I20260620 18:36:04.705588 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980527431848-22676-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980527431848-22676-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980527431848-22676-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980527431848-22676-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.37.1:0
--local_ip_for_outbound_sockets=127.22.37.1
--webserver_interface=127.22.37.1
--webserver_port=0
--tserver_master_addrs=127.22.37.62:32975
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.1.keytab
--principal=kudu/127.22.37.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:33595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:36:05.030076 26840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:36:05.030357 26840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:36:05.030488 26840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:36:05.037742 26840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:36:05.037851 26840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:36:05.038044 26840 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.37.1
I20260620 18:36:05.047618 26840 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:33595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980527431848-22676-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980527431848-22676-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.1.keytab
--principal=kudu/127.22.37.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.37.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980527431848-22676-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.37.1
--webserver_port=0
--tserver_master_addrs=127.22.37.62:32975
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.37.1
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980527431848-22676-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:36:05.048998 26840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:36:05.050446 26840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:36:05.063207 26845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:05.063251 26848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:05.063506 26846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:05.064220 26840 server_base.cc:1034] running on GCE node
I20260620 18:36:05.064828 26840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:36:05.065828 26840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:36:05.067018 26840 hybrid_clock.cc:648] HybridClock initialized: now 1781980565066977 us; error 31 us; skew 500 ppm
I20260620 18:36:05.067340 26840 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:36:05 dist-test-slave-5lgp krb5kdc[26735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980565, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:36:05.070449 26840 init.cc:377] Logged in from keytab as kudu/127.22.37.1@KRBTEST.COM (short username kudu)
I20260620 18:36:05.071517 26840 fs_manager.cc:359] Metadata directory not provided
I20260620 18:36:05.071625 26840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:36:05.072001 26840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:36:05.075076 26840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980527431848-22676-0/minicluster-data/ts-0/data/instance:
uuid: "692c9112662e487db0cf9c0cd37aa499"
format_stamp: "Formatted at 2026-06-20 18:36:05 on dist-test-slave-5lgp"
server_key: "3905cf9a9c2fbb16f9e2d3dd2a041cbf"
server_key_iv: "d60cff47a105ff370d4ee99d803bec91"
server_key_version: "encryptionkey@0"
I20260620 18:36:05.075848 26840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980527431848-22676-0/minicluster-data/ts-0/wal/instance:
uuid: "692c9112662e487db0cf9c0cd37aa499"
format_stamp: "Formatted at 2026-06-20 18:36:05 on dist-test-slave-5lgp"
server_key: "3905cf9a9c2fbb16f9e2d3dd2a041cbf"
server_key_iv: "d60cff47a105ff370d4ee99d803bec91"
server_key_version: "encryptionkey@0"
I20260620 18:36:05.080494 26840 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 18:36:05.083861 26854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:05.085100 26840 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:36:05.085274 26840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980527431848-22676-0/minicluster-data/ts-0/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980527431848-22676-0/minicluster-data/ts-0/wal
uuid: "692c9112662e487db0cf9c0cd37aa499"
format_stamp: "Formatted at 2026-06-20 18:36:05 on dist-test-slave-5lgp"
server_key: "3905cf9a9c2fbb16f9e2d3dd2a041cbf"
server_key_iv: "d60cff47a105ff370d4ee99d803bec91"
server_key_version: "encryptionkey@0"
I20260620 18:36:05.085520 26840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980527431848-22676-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980527431848-22676-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980527431848-22676-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:36:05.110159 26840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:36:05.114285 26840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:36:05.114598 26840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:36:05.115586 26840 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:36:05.116969 26840 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:36:05.117063 26840 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:05.117192 26840 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:36:05.117288 26840 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:05.154709 26840 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.1:36377
I20260620 18:36:05.154762 26966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.1:36377 every 8 connection(s)
I20260620 18:36:05.155534 26840 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980527431848-22676-0/minicluster-data/ts-0/data/info.pb
I20260620 18:36:05.164497 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 26840
I20260620 18:36:05.164711 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980527431848-22676-0/minicluster-data/ts-0/wal/instance
I20260620 18:36:05.165565 22676 external_mini_cluster.cc:1428] Setting key 132fe5b0b605913cd3c8f9f7002e3695
Jun 20 18:36:05 dist-test-slave-5lgp krb5kdc[26735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980565, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.1@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM
I20260620 18:36:05.176515 26969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:05.157950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.1:40687 (local address 127.22.37.62:32975)
0620 18:36:05.158362 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:05.158368 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:05.159405 (+ 1037us) server_negotiation.cc:407] Connection header received
0620 18:36:05.161078 (+ 1673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:05.161082 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:05.161144 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:05.161247 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:05.162680 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:05.163367 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:05.164952 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:05.165623 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:05.165764 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:05.169775 (+ 4011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:05.169784 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:05.169791 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:05.169835 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:05.173269 (+ 3434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:05.174280 (+ 1011us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:05.174285 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:05.174291 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:05.174402 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:05.174814 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:05.174825 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:05.174827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:05.175115 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:05.175264 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:05.175937 (+ 673us) server_negotiation.cc:299] Negotiation successful
0620 18:36:05.176143 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":200,"threads_started":1}
I20260620 18:36:05.177541 26970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:05.158569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:32975 (local address 127.22.37.1:40687)
0620 18:36:05.159232 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:05.159270 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:05.160849 (+ 1579us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:05.161493 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:05.161502 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:05.161986 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:05.162533 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:05.162549 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:05.163495 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:05.163499 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:05.164793 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:05.164813 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:05.165906 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:05.165910 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:05.166088 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:05.166864 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:05.166890 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:05.169574 (+ 2684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:36:05.173440 (+ 3866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:05.173451 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:05.173469 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:05.174155 (+ 686us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:05.174510 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:05.174517 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:05.174524 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:05.174705 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:05.175255 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:36:05.175262 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:36:05.175572 (+ 310us) client_negotiation.cc:769] Sending connection context
0620 18:36:05.175833 (+ 261us) client_negotiation.cc:240] Negotiation successful
0620 18:36:05.176354 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":262,"threads_started":1}
I20260620 18:36:05.179062 26967 heartbeater.cc:347] Connected to a master server at 127.22.37.62:32975
I20260620 18:36:05.179392 26967 heartbeater.cc:464] Registering TS with master...
I20260620 18:36:05.180142 26967 heartbeater.cc:511] Master 127.22.37.62:32975 requested a full tablet report, sending...
I20260620 18:36:05.182528 26779 ts_manager.cc:194] Registered new tserver with Master: 692c9112662e487db0cf9c0cd37aa499 (127.22.37.1:36377)
I20260620 18:36:05.184772 26779 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.37.1@KRBTEST.COM'} at 127.22.37.1:40687
WARNING: no policy specified for kudu/127.22.37.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.37.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.37.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.2.keytab.
Entry for principal kudu/127.22.37.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.2.keytab.
I20260620 18:36:05.210592 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980527431848-22676-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980527431848-22676-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980527431848-22676-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980527431848-22676-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.37.2:0
--local_ip_for_outbound_sockets=127.22.37.2
--webserver_interface=127.22.37.2
--webserver_port=0
--tserver_master_addrs=127.22.37.62:32975
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.2.keytab
--principal=kudu/127.22.37.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:33595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:36:05.525187 26973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:36:05.525585 26973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:36:05.525758 26973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:36:05.537210 26973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:36:05.537336 26973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:36:05.537524 26973 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.37.2
I20260620 18:36:05.550748 26973 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:33595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980527431848-22676-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980527431848-22676-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.2.keytab
--principal=kudu/127.22.37.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.37.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980527431848-22676-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.37.2
--webserver_port=0
--tserver_master_addrs=127.22.37.62:32975
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.37.2
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980527431848-22676-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:36:05.551987 26973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:36:05.553215 26973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:36:05.565935 26978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:05.565892 26981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:05.565973 26979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:05.567167 26973 server_base.cc:1034] running on GCE node
I20260620 18:36:05.567788 26973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:36:05.568735 26973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:36:05.569926 26973 hybrid_clock.cc:648] HybridClock initialized: now 1781980565569873 us; error 44 us; skew 500 ppm
I20260620 18:36:05.570240 26973 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:36:05 dist-test-slave-5lgp krb5kdc[26735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980565, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:36:05.573276 26973 init.cc:377] Logged in from keytab as kudu/127.22.37.2@KRBTEST.COM (short username kudu)
I20260620 18:36:05.574558 26973 fs_manager.cc:359] Metadata directory not provided
I20260620 18:36:05.574649 26973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:36:05.574954 26973 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:36:05.577857 26973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980527431848-22676-0/minicluster-data/ts-1/data/instance:
uuid: "06c0f151156347b987a1d38c50a0ee93"
format_stamp: "Formatted at 2026-06-20 18:36:05 on dist-test-slave-5lgp"
server_key: "d57e1f9d5f2985fd6f56c9d08c0d11cd"
server_key_iv: "6e0d37563964b12a5e6bec0a5afc5c39"
server_key_version: "encryptionkey@0"
I20260620 18:36:05.578568 26973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980527431848-22676-0/minicluster-data/ts-1/wal/instance:
uuid: "06c0f151156347b987a1d38c50a0ee93"
format_stamp: "Formatted at 2026-06-20 18:36:05 on dist-test-slave-5lgp"
server_key: "d57e1f9d5f2985fd6f56c9d08c0d11cd"
server_key_iv: "6e0d37563964b12a5e6bec0a5afc5c39"
server_key_version: "encryptionkey@0"
I20260620 18:36:05.583427 26973 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260620 18:36:05.586599 26987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:05.587918 26973 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:36:05.588078 26973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980527431848-22676-0/minicluster-data/ts-1/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980527431848-22676-0/minicluster-data/ts-1/wal
uuid: "06c0f151156347b987a1d38c50a0ee93"
format_stamp: "Formatted at 2026-06-20 18:36:05 on dist-test-slave-5lgp"
server_key: "d57e1f9d5f2985fd6f56c9d08c0d11cd"
server_key_iv: "6e0d37563964b12a5e6bec0a5afc5c39"
server_key_version: "encryptionkey@0"
I20260620 18:36:05.588279 26973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980527431848-22676-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980527431848-22676-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980527431848-22676-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:36:05.612986 26973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:36:05.617522 26973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:36:05.617810 26973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:36:05.618799 26973 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:36:05.620201 26973 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:36:05.620288 26973 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:05.620442 26973 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:36:05.620509 26973 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:05.652758 26973 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.2:35131
I20260620 18:36:05.652814 27099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.2:35131 every 8 connection(s)
I20260620 18:36:05.653688 26973 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980527431848-22676-0/minicluster-data/ts-1/data/info.pb
I20260620 18:36:05.658895 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 26973
I20260620 18:36:05.659108 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980527431848-22676-0/minicluster-data/ts-1/wal/instance
I20260620 18:36:05.659557 22676 external_mini_cluster.cc:1428] Setting key ff5435b77503afd7457ce3faa6273be7
Jun 20 18:36:05 dist-test-slave-5lgp krb5kdc[26735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980565, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.2@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM
I20260620 18:36:05.674893 26969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:05.656105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.2:34387 (local address 127.22.37.62:32975)
0620 18:36:05.656278 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:05.656283 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:05.657554 (+ 1271us) server_negotiation.cc:407] Connection header received
0620 18:36:05.659451 (+ 1897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:05.659456 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:05.659553 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:05.659665 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:05.661186 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:05.661965 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:05.663550 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:05.663989 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:05.664121 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:05.667730 (+ 3609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:05.667740 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:05.667748 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:05.667795 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:05.671376 (+ 3581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:05.672156 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:05.672165 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:05.672170 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:05.672328 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:05.672738 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:05.672746 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:05.672752 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:05.673153 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:05.673303 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:05.674122 (+ 819us) server_negotiation.cc:299] Negotiation successful
0620 18:36:05.674393 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 18:36:05.675767 27102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:05.656532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:32975 (local address 127.22.37.2:34387)
0620 18:36:05.657337 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:05.657440 (+ 103us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:05.659144 (+ 1704us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:05.659915 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:05.659924 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:05.660439 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:05.661011 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:05.661027 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:05.662117 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:05.662123 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:05.663398 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:05.663409 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:05.664228 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:05.664235 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:05.664459 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:05.665161 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:05.665194 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:05.667485 (+ 2291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:36:05.671543 (+ 4058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:05.671551 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:05.671566 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:05.672031 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:05.672434 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:05.672438 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:05.672440 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:05.672616 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:05.673272 (+ 656us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:36:05.673279 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:36:05.673704 (+ 425us) client_negotiation.cc:769] Sending connection context
0620 18:36:05.673985 (+ 281us) client_negotiation.cc:240] Negotiation successful
0620 18:36:05.674505 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":287,"threads_started":1}
I20260620 18:36:05.677434 27100 heartbeater.cc:347] Connected to a master server at 127.22.37.62:32975
I20260620 18:36:05.677752 27100 heartbeater.cc:464] Registering TS with master...
I20260620 18:36:05.678454 27100 heartbeater.cc:511] Master 127.22.37.62:32975 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.22.37.3@KRBTEST.COM; defaulting to no policy
I20260620 18:36:05.680006 26779 ts_manager.cc:194] Registered new tserver with Master: 06c0f151156347b987a1d38c50a0ee93 (127.22.37.2:35131)
I20260620 18:36:05.680811 26779 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.37.2@KRBTEST.COM'} at 127.22.37.2:34387
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.37.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.37.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.3.keytab.
Entry for principal kudu/127.22.37.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.3.keytab.
I20260620 18:36:05.704802 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980527431848-22676-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980527431848-22676-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980527431848-22676-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980527431848-22676-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.37.3:0
--local_ip_for_outbound_sockets=127.22.37.3
--webserver_interface=127.22.37.3
--webserver_port=0
--tserver_master_addrs=127.22.37.62:32975
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.3.keytab
--principal=kudu/127.22.37.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:33595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:36:06.021389 27105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:36:06.021721 27105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:36:06.021891 27105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:36:06.028939 27105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:36:06.029089 27105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:36:06.029258 27105 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.37.3
I20260620 18:36:06.038859 27105 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:33595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980527431848-22676-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980527431848-22676-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.3.keytab
--principal=kudu/127.22.37.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.37.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980527431848-22676-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.37.3
--webserver_port=0
--tserver_master_addrs=127.22.37.62:32975
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.37.3
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980527431848-22676-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:36:06.040141 27105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:36:06.041494 27105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:36:06.054328 27111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:06.054392 27110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:06.054373 27113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:06.055018 27105 server_base.cc:1034] running on GCE node
I20260620 18:36:06.055657 27105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:36:06.056591 27105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:36:06.057830 27105 hybrid_clock.cc:648] HybridClock initialized: now 1781980566057775 us; error 44 us; skew 500 ppm
I20260620 18:36:06.058248 27105 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:36:06 dist-test-slave-5lgp krb5kdc[26735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980566, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:36:06.061283 27105 init.cc:377] Logged in from keytab as kudu/127.22.37.3@KRBTEST.COM (short username kudu)
I20260620 18:36:06.062505 27105 fs_manager.cc:359] Metadata directory not provided
I20260620 18:36:06.062613 27105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:36:06.062934 27105 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:36:06.065856 27105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980527431848-22676-0/minicluster-data/ts-2/data/instance:
uuid: "49b45ea90f354aebb8a8801be1222b03"
format_stamp: "Formatted at 2026-06-20 18:36:06 on dist-test-slave-5lgp"
server_key: "6ca38c91dffc8327a8b4737004716e79"
server_key_iv: "87bc25bcfb28eb7abde446ebea52f342"
server_key_version: "encryptionkey@0"
I20260620 18:36:06.066656 27105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980527431848-22676-0/minicluster-data/ts-2/wal/instance:
uuid: "49b45ea90f354aebb8a8801be1222b03"
format_stamp: "Formatted at 2026-06-20 18:36:06 on dist-test-slave-5lgp"
server_key: "6ca38c91dffc8327a8b4737004716e79"
server_key_iv: "87bc25bcfb28eb7abde446ebea52f342"
server_key_version: "encryptionkey@0"
I20260620 18:36:06.071604 27105 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:36:06.075012 27119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:06.076227 27105 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 18:36:06.076381 27105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980527431848-22676-0/minicluster-data/ts-2/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980527431848-22676-0/minicluster-data/ts-2/wal
uuid: "49b45ea90f354aebb8a8801be1222b03"
format_stamp: "Formatted at 2026-06-20 18:36:06 on dist-test-slave-5lgp"
server_key: "6ca38c91dffc8327a8b4737004716e79"
server_key_iv: "87bc25bcfb28eb7abde446ebea52f342"
server_key_version: "encryptionkey@0"
I20260620 18:36:06.076582 27105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980527431848-22676-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980527431848-22676-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980527431848-22676-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:36:06.111176 27105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:36:06.115018 27105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:36:06.115309 27105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:36:06.116288 27105 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:36:06.117571 27105 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:36:06.117679 27105 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:06.117799 27105 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:36:06.117884 27105 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:06.150164 27105 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.3:37705
I20260620 18:36:06.150249 27231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.3:37705 every 8 connection(s)
I20260620 18:36:06.151010 27105 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980527431848-22676-0/minicluster-data/ts-2/data/info.pb
I20260620 18:36:06.153638 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 27105
I20260620 18:36:06.153843 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980527431848-22676-0/minicluster-data/ts-2/wal/instance
I20260620 18:36:06.154421 22676 external_mini_cluster.cc:1428] Setting key 4689a6bbf5d6a90d829e595a2e5b4453
Jun 20 18:36:06 dist-test-slave-5lgp krb5kdc[26735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980566, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.3@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM
I20260620 18:36:06.171255 26969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:06.153528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.3:43981 (local address 127.22.37.62:32975)
0620 18:36:06.153767 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:06.153774 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:06.155235 (+ 1461us) server_negotiation.cc:407] Connection header received
0620 18:36:06.157492 (+ 2257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:06.157497 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:06.157598 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:06.157705 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:06.159596 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:06.160318 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:06.161901 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:06.162318 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:06.162413 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:06.165682 (+ 3269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:06.165687 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:06.165690 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:06.165720 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:06.168215 (+ 2495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:06.169019 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:06.169025 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:06.169029 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:06.169159 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:06.169609 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:06.169614 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:06.169618 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:06.169913 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:06.170067 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:06.170715 (+ 648us) server_negotiation.cc:299] Negotiation successful
0620 18:36:06.170915 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260620 18:36:06.172115 27234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:06.154138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:32975 (local address 127.22.37.3:43981)
0620 18:36:06.155076 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:06.155128 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:06.157159 (+ 2031us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:06.157957 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:06.157969 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:06.158678 (+ 709us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:06.159416 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:06.159443 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:06.160452 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:06.160456 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:06.161769 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:06.161780 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:06.162521 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:06.162527 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:06.162724 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:06.163415 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:06.163436 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:06.165517 (+ 2081us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:36:06.168345 (+ 2828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:06.168354 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:06.168368 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:06.168870 (+ 502us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:06.169266 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:06.169272 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:06.169275 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:06.169500 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:06.170046 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:36:06.170053 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:36:06.170367 (+ 314us) client_negotiation.cc:769] Sending connection context
0620 18:36:06.170613 (+ 246us) client_negotiation.cc:240] Negotiation successful
0620 18:36:06.171092 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":666,"thread_start_us":462,"threads_started":1}
I20260620 18:36:06.173344 27232 heartbeater.cc:347] Connected to a master server at 127.22.37.62:32975
I20260620 18:36:06.173678 27232 heartbeater.cc:464] Registering TS with master...
I20260620 18:36:06.174294 27232 heartbeater.cc:511] Master 127.22.37.62:32975 requested a full tablet report, sending...
I20260620 18:36:06.175602 26779 ts_manager.cc:194] Registered new tserver with Master: 49b45ea90f354aebb8a8801be1222b03 (127.22.37.3:37705)
I20260620 18:36:06.176398 26779 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.37.3@KRBTEST.COM'} at 127.22.37.3:43981
I20260620 18:36:06.182080 22676 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:36:06.187914 26967 heartbeater.cc:503] Master 127.22.37.62:32975 was elected leader, sending a full tablet report...
I20260620 18:36:06.199040 26969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:06.186487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48094 (local address 127.22.37.62:32975)
0620 18:36:06.186699 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:06.186708 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:06.187111 (+ 403us) server_negotiation.cc:407] Connection header received
0620 18:36:06.187243 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:06.187246 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:06.187301 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:06.187378 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:06.188913 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:06.189923 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:06.191138 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:06.191649 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:06.191787 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:06.194208 (+ 2421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:06.194213 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:06.194216 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:06.194243 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:06.196708 (+ 2465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:06.197356 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:06.197360 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:06.197362 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:06.197535 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:06.197843 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:06.197846 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:06.197848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:06.198115 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:06.198291 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:06.198552 (+ 261us) server_negotiation.cc:299] Negotiation successful
0620 18:36:06.198706 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 18:36:06.202533 26779 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48094:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:36:06.209025 26779 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:36:06.231966 27245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:06.220784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32954 (local address 127.22.37.2:35131)
0620 18:36:06.222915 (+ 2131us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:06.222925 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:06.222958 (+ 33us) server_negotiation.cc:407] Connection header received
0620 18:36:06.223065 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:06.223074 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:06.223334 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:06.225272 (+ 1938us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:36:06.225500 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:06.226921 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:06.229822 (+ 2901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:06.230869 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:06.231037 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:06.231284 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:06.231488 (+ 204us) server_negotiation.cc:299] Negotiation successful
0620 18:36:06.231643 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1996,"thread_start_us":1913,"threads_started":1}
I20260620 18:36:06.231993 27246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:06.220542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.2:35131 (local address 127.0.0.1:32954)
0620 18:36:06.222109 (+ 1567us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:06.222163 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:06.222379 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:06.223591 (+ 1212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:06.223602 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:06.223632 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:36:06.224042 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:06.224054 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:06.227054 (+ 3000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:06.227059 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:06.229674 (+ 2615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:06.229685 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:06.231050 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:06.231055 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:06.231249 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:06.231306 (+ 57us) client_negotiation.cc:769] Sending connection context
0620 18:36:06.231450 (+ 144us) client_negotiation.cc:240] Negotiation successful
0620 18:36:06.231641 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1261,"thread_start_us":645,"threads_started":1}
I20260620 18:36:06.233165 27244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:06.221173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.3:37705 (local address 127.0.0.1:58014)
0620 18:36:06.223542 (+ 2369us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:06.223559 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:06.223663 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:06.224545 (+ 882us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:06.224549 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:06.224572 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:36:06.225015 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:06.225027 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:06.227039 (+ 2012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:06.227044 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:06.230074 (+ 3030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:06.230087 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:06.232271 (+ 2184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:06.232277 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:06.232480 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:06.232511 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 18:36:06.232601 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 18:36:06.232733 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2241,"spinlock_wait_cycles":2944,"thread_start_us":736,"threads_started":1}
I20260620 18:36:06.234009 27247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:06.221677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58014 (local address 127.22.37.3:37705)
0620 18:36:06.223280 (+ 1603us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:06.223287 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:06.223799 (+ 512us) server_negotiation.cc:407] Connection header received
0620 18:36:06.223890 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:06.223900 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:06.224124 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:06.224293 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:36:06.225211 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:06.226881 (+ 1670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:06.230240 (+ 3359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:06.231712 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:06.231836 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:06.232045 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:06.233508 (+ 1463us) server_negotiation.cc:299] Negotiation successful
0620 18:36:06.233671 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1465,"thread_start_us":1336,"threads_started":1}
I20260620 18:36:06.235710 27035 tablet_service.cc:1467] Processing CreateTablet for tablet 650dc9a697b44fdfa565b6dbbbceff56 (DEFAULT_TABLE table=test-table [id=4ee7da6950ca499e8a91ef3bfe93dd10]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:36:06.236800 27248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:06.222047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.1:36377 (local address 127.0.0.1:38068)
0620 18:36:06.228247 (+ 6200us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:06.228264 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:06.228388 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:06.229216 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:06.229221 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:06.229242 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:36:06.229653 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:06.229664 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:06.231514 (+ 1850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:06.231521 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:06.234411 (+ 2890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:06.234421 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:06.236018 (+ 1597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:06.236024 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:06.236200 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:06.236230 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 18:36:06.236371 (+ 141us) client_negotiation.cc:240] Negotiation successful
0620 18:36:06.236498 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6076,"thread_start_us":5939,"threads_started":1}
I20260620 18:36:06.236829 27249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:06.222417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38068 (local address 127.22.37.1:36377)
0620 18:36:06.227860 (+ 5443us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:06.227868 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:06.228507 (+ 639us) server_negotiation.cc:407] Connection header received
0620 18:36:06.228592 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:06.228599 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:06.228826 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:06.228982 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:36:06.229853 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:06.231349 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:06.234552 (+ 3203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:06.235824 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:06.235950 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:06.236140 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:06.236387 (+ 247us) server_negotiation.cc:299] Negotiation successful
0620 18:36:06.236532 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5247,"thread_start_us":5100,"threads_started":1}
I20260620 18:36:06.237152 27035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 650dc9a697b44fdfa565b6dbbbceff56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:36:06.237489 27167 tablet_service.cc:1467] Processing CreateTablet for tablet 650dc9a697b44fdfa565b6dbbbceff56 (DEFAULT_TABLE table=test-table [id=4ee7da6950ca499e8a91ef3bfe93dd10]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:36:06.238816 27167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 650dc9a697b44fdfa565b6dbbbceff56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:36:06.240435 26902 tablet_service.cc:1467] Processing CreateTablet for tablet 650dc9a697b44fdfa565b6dbbbceff56 (DEFAULT_TABLE table=test-table [id=4ee7da6950ca499e8a91ef3bfe93dd10]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:36:06.241672 26902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 650dc9a697b44fdfa565b6dbbbceff56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:36:06.248072 27250 tablet_bootstrap.cc:492] T 650dc9a697b44fdfa565b6dbbbceff56 P 06c0f151156347b987a1d38c50a0ee93: Bootstrap starting.
I20260620 18:36:06.251363 27250 tablet_bootstrap.cc:654] T 650dc9a697b44fdfa565b6dbbbceff56 P 06c0f151156347b987a1d38c50a0ee93: Neither blocks nor log segments found. Creating new log.
I20260620 18:36:06.252308 27250 log.cc:824] T 650dc9a697b44fdfa565b6dbbbceff56 P 06c0f151156347b987a1d38c50a0ee93: Log is configured to *not* fsync() on all Append() calls
I20260620 18:36:06.254223 27252 tablet_bootstrap.cc:492] T 650dc9a697b44fdfa565b6dbbbceff56 P 49b45ea90f354aebb8a8801be1222b03: Bootstrap starting.
I20260620 18:36:06.255267 27253 tablet_bootstrap.cc:492] T 650dc9a697b44fdfa565b6dbbbceff56 P 692c9112662e487db0cf9c0cd37aa499: Bootstrap starting.
I20260620 18:36:06.255303 27250 tablet_bootstrap.cc:492] T 650dc9a697b44fdfa565b6dbbbceff56 P 06c0f151156347b987a1d38c50a0ee93: No bootstrap required, opened a new log
I20260620 18:36:06.255565 27250 ts_tablet_manager.cc:1397] T 650dc9a697b44fdfa565b6dbbbceff56 P 06c0f151156347b987a1d38c50a0ee93: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260620 18:36:06.257591 27252 tablet_bootstrap.cc:654] T 650dc9a697b44fdfa565b6dbbbceff56 P 49b45ea90f354aebb8a8801be1222b03: Neither blocks nor log segments found. Creating new log.
I20260620 18:36:06.258014 27253 tablet_bootstrap.cc:654] T 650dc9a697b44fdfa565b6dbbbceff56 P 692c9112662e487db0cf9c0cd37aa499: Neither blocks nor log segments found. Creating new log.
I20260620 18:36:06.259151 27252 log.cc:824] T 650dc9a697b44fdfa565b6dbbbceff56 P 49b45ea90f354aebb8a8801be1222b03: Log is configured to *not* fsync() on all Append() calls
I20260620 18:36:06.259243 27253 log.cc:824] T 650dc9a697b44fdfa565b6dbbbceff56 P 692c9112662e487db0cf9c0cd37aa499: Log is configured to *not* fsync() on all Append() calls
I20260620 18:36:06.262090 27252 tablet_bootstrap.cc:492] T 650dc9a697b44fdfa565b6dbbbceff56 P 49b45ea90f354aebb8a8801be1222b03: No bootstrap required, opened a new log
I20260620 18:36:06.262351 27252 ts_tablet_manager.cc:1397] T 650dc9a697b44fdfa565b6dbbbceff56 P 49b45ea90f354aebb8a8801be1222b03: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.001s
I20260620 18:36:06.262363 27253 tablet_bootstrap.cc:492] T 650dc9a697b44fdfa565b6dbbbceff56 P 692c9112662e487db0cf9c0cd37aa499: No bootstrap required, opened a new log
I20260620 18:36:06.262610 27253 ts_tablet_manager.cc:1397] T 650dc9a697b44fdfa565b6dbbbceff56 P 692c9112662e487db0cf9c0cd37aa499: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.001s
I20260620 18:36:06.265911 27250 raft_consensus.cc:348] T 650dc9a697b44fdfa565b6dbbbceff56 P 06c0f151156347b987a1d38c50a0ee93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06c0f151156347b987a1d38c50a0ee93" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 35131 } } peers { permanent_uuid: "692c9112662e487db0cf9c0cd37aa499" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 36377 } } peers { permanent_uuid: "49b45ea90f354aebb8a8801be1222b03" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 37705 } }
I20260620 18:36:06.266360 27250 raft_consensus.cc:374] T 650dc9a697b44fdfa565b6dbbbceff56 P 06c0f151156347b987a1d38c50a0ee93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:36:06.266477 27250 raft_consensus.cc:729] T 650dc9a697b44fdfa565b6dbbbceff56 P 06c0f151156347b987a1d38c50a0ee93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06c0f151156347b987a1d38c50a0ee93, State: Initialized, Role: FOLLOWER
I20260620 18:36:06.267129 27250 consensus_queue.cc:260] T 650dc9a697b44fdfa565b6dbbbceff56 P 06c0f151156347b987a1d38c50a0ee93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06c0f151156347b987a1d38c50a0ee93" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 35131 } } peers { permanent_uuid: "692c9112662e487db0cf9c0cd37aa499" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 36377 } } peers { permanent_uuid: "49b45ea90f354aebb8a8801be1222b03" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 37705 } }
I20260620 18:36:06.268257 27100 heartbeater.cc:503] Master 127.22.37.62:32975 was elected leader, sending a full tablet report...
I20260620 18:36:06.268988 27250 ts_tablet_manager.cc:1428] T 650dc9a697b44fdfa565b6dbbbceff56 P 06c0f151156347b987a1d38c50a0ee93: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260620 18:36:06.270318 27252 raft_consensus.cc:348] T 650dc9a697b44fdfa565b6dbbbceff56 P 49b45ea90f354aebb8a8801be1222b03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06c0f151156347b987a1d38c50a0ee93" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 35131 } } peers { permanent_uuid: "692c9112662e487db0cf9c0cd37aa499" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 36377 } } peers { permanent_uuid: "49b45ea90f354aebb8a8801be1222b03" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 37705 } }
I20260620 18:36:06.270704 27252 raft_consensus.cc:374] T 650dc9a697b44fdfa565b6dbbbceff56 P 49b45ea90f354aebb8a8801be1222b03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:36:06.270799 27252 raft_consensus.cc:729] T 650dc9a697b44fdfa565b6dbbbceff56 P 49b45ea90f354aebb8a8801be1222b03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49b45ea90f354aebb8a8801be1222b03, State: Initialized, Role: FOLLOWER
I20260620 18:36:06.271432 27252 consensus_queue.cc:260] T 650dc9a697b44fdfa565b6dbbbceff56 P 49b45ea90f354aebb8a8801be1222b03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06c0f151156347b987a1d38c50a0ee93" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 35131 } } peers { permanent_uuid: "692c9112662e487db0cf9c0cd37aa499" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 36377 } } peers { permanent_uuid: "49b45ea90f354aebb8a8801be1222b03" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 37705 } }
I20260620 18:36:06.272454 27232 heartbeater.cc:503] Master 127.22.37.62:32975 was elected leader, sending a full tablet report...
I20260620 18:36:06.273154 27252 ts_tablet_manager.cc:1428] T 650dc9a697b44fdfa565b6dbbbceff56 P 49b45ea90f354aebb8a8801be1222b03: Time spent starting tablet: real 0.011s user 0.008s sys 0.003s
I20260620 18:36:06.272948 27253 raft_consensus.cc:348] T 650dc9a697b44fdfa565b6dbbbceff56 P 692c9112662e487db0cf9c0cd37aa499 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06c0f151156347b987a1d38c50a0ee93" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 35131 } } peers { permanent_uuid: "692c9112662e487db0cf9c0cd37aa499" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 36377 } } peers { permanent_uuid: "49b45ea90f354aebb8a8801be1222b03" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 37705 } }
I20260620 18:36:06.273370 27253 raft_consensus.cc:374] T 650dc9a697b44fdfa565b6dbbbceff56 P 692c9112662e487db0cf9c0cd37aa499 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:36:06.273509 27253 raft_consensus.cc:729] T 650dc9a697b44fdfa565b6dbbbceff56 P 692c9112662e487db0cf9c0cd37aa499 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 692c9112662e487db0cf9c0cd37aa499, State: Initialized, Role: FOLLOWER
I20260620 18:36:06.274049 27253 consensus_queue.cc:260] T 650dc9a697b44fdfa565b6dbbbceff56 P 692c9112662e487db0cf9c0cd37aa499 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06c0f151156347b987a1d38c50a0ee93" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 35131 } } peers { permanent_uuid: "692c9112662e487db0cf9c0cd37aa499" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 36377 } } peers { permanent_uuid: "49b45ea90f354aebb8a8801be1222b03" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 37705 } }
I20260620 18:36:06.275749 27253 ts_tablet_manager.cc:1428] T 650dc9a697b44fdfa565b6dbbbceff56 P 692c9112662e487db0cf9c0cd37aa499: Time spent starting tablet: real 0.013s user 0.004s sys 0.009s
W20260620 18:36:06.402556 27233 tablet.cc:2369] T 650dc9a697b44fdfa565b6dbbbceff56 P 49b45ea90f354aebb8a8801be1222b03: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:36:06.405951 27101 tablet.cc:2369] T 650dc9a697b44fdfa565b6dbbbceff56 P 06c0f151156347b987a1d38c50a0ee93: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:36:06.407790 26968 tablet.cc:2369] T 650dc9a697b44fdfa565b6dbbbceff56 P 692c9112662e487db0cf9c0cd37aa499: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:36:06.661556 27257 raft_consensus.cc:482] T 650dc9a697b44fdfa565b6dbbbceff56 P 49b45ea90f354aebb8a8801be1222b03 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:36:06.661841 27257 raft_consensus.cc:504] T 650dc9a697b44fdfa565b6dbbbceff56 P 49b45ea90f354aebb8a8801be1222b03 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06c0f151156347b987a1d38c50a0ee93" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 35131 } } peers { permanent_uuid: "692c9112662e487db0cf9c0cd37aa499" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 36377 } } peers { permanent_uuid: "49b45ea90f354aebb8a8801be1222b03" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 37705 } }
I20260620 18:36:06.663363 27257 leader_election.cc:288] T 650dc9a697b44fdfa565b6dbbbceff56 P 49b45ea90f354aebb8a8801be1222b03 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06c0f151156347b987a1d38c50a0ee93 (127.22.37.2:35131), 692c9112662e487db0cf9c0cd37aa499 (127.22.37.1:36377)
I20260620 18:36:06.668960 27234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:06.663767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.2:35131 (local address 127.22.37.3:54459)
0620 18:36:06.663967 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:06.663984 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:06.664114 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:06.664626 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:06.664630 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:06.664650 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:36:06.664793 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:06.664803 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:06.665813 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:06.665818 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:06.667061 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:06.667072 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:06.668207 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:06.668212 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:06.668400 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:06.668422 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 18:36:06.668527 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 18:36:06.668679 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260620 18:36:06.668960 27245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:06.663855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.3:54459 (local address 127.22.37.2:35131)
0620 18:36:06.664068 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:06.664074 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:06.664095 (+ 21us) server_negotiation.cc:407] Connection header received
0620 18:36:06.664287 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:06.664290 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:06.664340 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:06.664444 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:36:06.664929 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:06.665679 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:06.667226 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:06.668052 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:06.668199 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:06.668316 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:06.668550 (+ 234us) server_negotiation.cc:299] Negotiation successful
0620 18:36:06.668679 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 18:36:06.669850 27249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:06.664330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.3:34613 (local address 127.22.37.1:36377)
0620 18:36:06.664538 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:06.664545 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:06.664886 (+ 341us) server_negotiation.cc:407] Connection header received
0620 18:36:06.665066 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:06.665070 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:06.665124 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:06.665215 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:36:06.665733 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:06.666413 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:06.668036 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:06.668935 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:06.669079 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:06.669169 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:06.669515 (+ 346us) server_negotiation.cc:299] Negotiation successful
0620 18:36:06.669628 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 18:36:06.669910 27259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:06.664154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.1:36377 (local address 127.22.37.3:34613)
0620 18:36:06.664792 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:06.664805 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:06.664934 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:06.665227 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:06.665232 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:06.665254 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:36:06.665594 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:06.665602 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:06.666558 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:06.666564 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:06.667875 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:06.667884 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:06.669118 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:06.669124 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:06.669297 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:06.669334 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 18:36:06.669483 (+ 149us) client_negotiation.cc:240] Negotiation successful
0620 18:36:06.669600 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":362,"threads_started":1}
I20260620 18:36:06.670063 27055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "650dc9a697b44fdfa565b6dbbbceff56" candidate_uuid: "49b45ea90f354aebb8a8801be1222b03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06c0f151156347b987a1d38c50a0ee93" is_pre_election: true
I20260620 18:36:06.670648 27055 raft_consensus.cc:2454] T 650dc9a697b44fdfa565b6dbbbceff56 P 06c0f151156347b987a1d38c50a0ee93 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49b45ea90f354aebb8a8801be1222b03 in term 0.
I20260620 18:36:06.670821 26922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "650dc9a697b44fdfa565b6dbbbceff56" candidate_uuid: "49b45ea90f354aebb8a8801be1222b03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "692c9112662e487db0cf9c0cd37aa499" is_pre_election: true
I20260620 18:36:06.671342 26922 raft_consensus.cc:2454] T 650dc9a697b44fdfa565b6dbbbceff56 P 692c9112662e487db0cf9c0cd37aa499 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49b45ea90f354aebb8a8801be1222b03 in term 0.
I20260620 18:36:06.671362 27123 leader_election.cc:302] T 650dc9a697b44fdfa565b6dbbbceff56 P 49b45ea90f354aebb8a8801be1222b03 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06c0f151156347b987a1d38c50a0ee93, 49b45ea90f354aebb8a8801be1222b03; no voters:
I20260620 18:36:06.671731 27257 raft_consensus.cc:2781] T 650dc9a697b44fdfa565b6dbbbceff56 P 49b45ea90f354aebb8a8801be1222b03 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:36:06.671890 27257 raft_consensus.cc:482] T 650dc9a697b44fdfa565b6dbbbceff56 P 49b45ea90f354aebb8a8801be1222b03 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:36:06.672011 27257 raft_consensus.cc:3037] T 650dc9a697b44fdfa565b6dbbbceff56 P 49b45ea90f354aebb8a8801be1222b03 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:36:06.674269 27257 raft_consensus.cc:504] T 650dc9a697b44fdfa565b6dbbbceff56 P 49b45ea90f354aebb8a8801be1222b03 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06c0f151156347b987a1d38c50a0ee93" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 35131 } } peers { permanent_uuid: "692c9112662e487db0cf9c0cd37aa499" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 36377 } } peers { permanent_uuid: "49b45ea90f354aebb8a8801be1222b03" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 37705 } }
I20260620 18:36:06.674885 27257 leader_election.cc:288] T 650dc9a697b44fdfa565b6dbbbceff56 P 49b45ea90f354aebb8a8801be1222b03 [CANDIDATE]: Term 1 election: Requested vote from peers 06c0f151156347b987a1d38c50a0ee93 (127.22.37.2:35131), 692c9112662e487db0cf9c0cd37aa499 (127.22.37.1:36377)
I20260620 18:36:06.675345 27055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "650dc9a697b44fdfa565b6dbbbceff56" candidate_uuid: "49b45ea90f354aebb8a8801be1222b03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06c0f151156347b987a1d38c50a0ee93"
I20260620 18:36:06.675410 26922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "650dc9a697b44fdfa565b6dbbbceff56" candidate_uuid: "49b45ea90f354aebb8a8801be1222b03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "692c9112662e487db0cf9c0cd37aa499"
I20260620 18:36:06.675534 27055 raft_consensus.cc:3037] T 650dc9a697b44fdfa565b6dbbbceff56 P 06c0f151156347b987a1d38c50a0ee93 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:36:06.675658 26922 raft_consensus.cc:3037] T 650dc9a697b44fdfa565b6dbbbceff56 P 692c9112662e487db0cf9c0cd37aa499 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:36:06.677820 27055 raft_consensus.cc:2454] T 650dc9a697b44fdfa565b6dbbbceff56 P 06c0f151156347b987a1d38c50a0ee93 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49b45ea90f354aebb8a8801be1222b03 in term 1.
I20260620 18:36:06.677834 26922 raft_consensus.cc:2454] T 650dc9a697b44fdfa565b6dbbbceff56 P 692c9112662e487db0cf9c0cd37aa499 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49b45ea90f354aebb8a8801be1222b03 in term 1.
I20260620 18:36:06.678200 27123 leader_election.cc:302] T 650dc9a697b44fdfa565b6dbbbceff56 P 49b45ea90f354aebb8a8801be1222b03 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06c0f151156347b987a1d38c50a0ee93, 49b45ea90f354aebb8a8801be1222b03; no voters:
I20260620 18:36:06.678496 27257 raft_consensus.cc:2781] T 650dc9a697b44fdfa565b6dbbbceff56 P 49b45ea90f354aebb8a8801be1222b03 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:36:06.678920 27257 raft_consensus.cc:686] T 650dc9a697b44fdfa565b6dbbbceff56 P 49b45ea90f354aebb8a8801be1222b03 [term 1 LEADER]: Becoming Leader. State: Replica: 49b45ea90f354aebb8a8801be1222b03, State: Running, Role: LEADER
I20260620 18:36:06.679747 27257 consensus_queue.cc:237] T 650dc9a697b44fdfa565b6dbbbceff56 P 49b45ea90f354aebb8a8801be1222b03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06c0f151156347b987a1d38c50a0ee93" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 35131 } } peers { permanent_uuid: "692c9112662e487db0cf9c0cd37aa499" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 36377 } } peers { permanent_uuid: "49b45ea90f354aebb8a8801be1222b03" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 37705 } }
I20260620 18:36:06.684432 26779 catalog_manager.cc:5495] T 650dc9a697b44fdfa565b6dbbbceff56 P 49b45ea90f354aebb8a8801be1222b03 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49b45ea90f354aebb8a8801be1222b03 (127.22.37.3). New cstate: current_term: 1 leader_uuid: "49b45ea90f354aebb8a8801be1222b03" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06c0f151156347b987a1d38c50a0ee93" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 35131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "692c9112662e487db0cf9c0cd37aa499" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 36377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49b45ea90f354aebb8a8801be1222b03" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 37705 } health_report { overall_health: HEALTHY } } }
I20260620 18:36:06.794764 27263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:06.788623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58028 (local address 127.22.37.3:37705)
0620 18:36:06.789041 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:06.789045 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:06.789539 (+ 494us) server_negotiation.cc:407] Connection header received
0620 18:36:06.789805 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:06.789808 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:06.789899 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:06.790029 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:36:06.790469 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:06.791224 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:06.792780 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:06.793234 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:06.793329 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:06.793802 (+ 473us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:36:06.794306 (+ 504us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:36:06.794411 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:06.794480 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 18:36:06.794540 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":201,"threads_started":1}
I20260620 18:36:06.802284 26922 raft_consensus.cc:1261] T 650dc9a697b44fdfa565b6dbbbceff56 P 692c9112662e487db0cf9c0cd37aa499 [term 1 FOLLOWER]: Refusing update from remote peer 49b45ea90f354aebb8a8801be1222b03: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:36:06.802301 27055 raft_consensus.cc:1261] T 650dc9a697b44fdfa565b6dbbbceff56 P 06c0f151156347b987a1d38c50a0ee93 [term 1 FOLLOWER]: Refusing update from remote peer 49b45ea90f354aebb8a8801be1222b03: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:36:06.803169 27260 consensus_queue.cc:1035] T 650dc9a697b44fdfa565b6dbbbceff56 P 49b45ea90f354aebb8a8801be1222b03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c0f151156347b987a1d38c50a0ee93" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:36:06.803530 27257 consensus_queue.cc:1035] T 650dc9a697b44fdfa565b6dbbbceff56 P 49b45ea90f354aebb8a8801be1222b03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "692c9112662e487db0cf9c0cd37aa499" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 36377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:36:06.817044 27268 mvcc.cc:205] Tried to move back new op lower bound from 7298992401612877824 to 7298992401124323328. Current Snapshot: MvccSnapshot[applied={T|T < 7298992401612877824}]
I20260620 18:36:06.830260 26779 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48094:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:36:06.830526 26779 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48094:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:36:06.833693 26779 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dd43eb11ef5f4047bc4d9ef81617990f: Sending DeleteTablet for 3 replicas of tablet 650dc9a697b44fdfa565b6dbbbceff56
I20260620 18:36:06.834725 27035 tablet_service.cc:1514] Processing DeleteTablet for tablet 650dc9a697b44fdfa565b6dbbbceff56 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:36:06 UTC) from {username='kudu', principal='kudu/127.22.37.62@KRBTEST.COM'} at 127.0.0.1:32954
I20260620 18:36:06.834908 27167 tablet_service.cc:1514] Processing DeleteTablet for tablet 650dc9a697b44fdfa565b6dbbbceff56 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:36:06 UTC) from {username='kudu', principal='kudu/127.22.37.62@KRBTEST.COM'} at 127.0.0.1:58014
I20260620 18:36:06.834908 26902 tablet_service.cc:1514] Processing DeleteTablet for tablet 650dc9a697b44fdfa565b6dbbbceff56 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:36:06 UTC) from {username='kudu', principal='kudu/127.22.37.62@KRBTEST.COM'} at 127.0.0.1:38068
I20260620 18:36:06.836952 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 26840
I20260620 18:36:06.848223 27277 tablet_replica.cc:330] T 650dc9a697b44fdfa565b6dbbbceff56 P 06c0f151156347b987a1d38c50a0ee93: stopping tablet replica
I20260620 18:36:06.848937 27277 raft_consensus.cc:2229] T 650dc9a697b44fdfa565b6dbbbceff56 P 06c0f151156347b987a1d38c50a0ee93 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:36:06.849706 27277 raft_consensus.cc:2258] T 650dc9a697b44fdfa565b6dbbbceff56 P 06c0f151156347b987a1d38c50a0ee93 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:36:06.852161 27277 ts_tablet_manager.cc:1905] T 650dc9a697b44fdfa565b6dbbbceff56 P 06c0f151156347b987a1d38c50a0ee93: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:36:06.852865 27278 tablet_replica.cc:330] T 650dc9a697b44fdfa565b6dbbbceff56 P 49b45ea90f354aebb8a8801be1222b03: stopping tablet replica
I20260620 18:36:06.853360 27278 raft_consensus.cc:2229] T 650dc9a697b44fdfa565b6dbbbceff56 P 49b45ea90f354aebb8a8801be1222b03 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:36:06.853806 27279 tablet_replica.cc:330] T 650dc9a697b44fdfa565b6dbbbceff56 P 692c9112662e487db0cf9c0cd37aa499: stopping tablet replica
I20260620 18:36:06.854176 27279 raft_consensus.cc:2229] T 650dc9a697b44fdfa565b6dbbbceff56 P 692c9112662e487db0cf9c0cd37aa499 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:36:06.854640 27279 raft_consensus.cc:2258] T 650dc9a697b44fdfa565b6dbbbceff56 P 692c9112662e487db0cf9c0cd37aa499 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:36:06.855050 27278 raft_consensus.cc:2258] T 650dc9a697b44fdfa565b6dbbbceff56 P 49b45ea90f354aebb8a8801be1222b03 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:36:06.856405 27279 ts_tablet_manager.cc:1905] T 650dc9a697b44fdfa565b6dbbbceff56 P 692c9112662e487db0cf9c0cd37aa499: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:36:06.858129 27278 ts_tablet_manager.cc:1905] T 650dc9a697b44fdfa565b6dbbbceff56 P 49b45ea90f354aebb8a8801be1222b03: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:36:06.863626 27279 ts_tablet_manager.cc:1918] T 650dc9a697b44fdfa565b6dbbbceff56 P 692c9112662e487db0cf9c0cd37aa499: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:36:06.863803 27279 log.cc:1196] T 650dc9a697b44fdfa565b6dbbbceff56 P 692c9112662e487db0cf9c0cd37aa499: Deleting WAL directory at /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980527431848-22676-0/minicluster-data/ts-0/wal/wals/650dc9a697b44fdfa565b6dbbbceff56
I20260620 18:36:06.864106 27277 ts_tablet_manager.cc:1918] T 650dc9a697b44fdfa565b6dbbbceff56 P 06c0f151156347b987a1d38c50a0ee93: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:36:06.864290 27279 ts_tablet_manager.cc:1939] T 650dc9a697b44fdfa565b6dbbbceff56 P 692c9112662e487db0cf9c0cd37aa499: Deleting consensus metadata
I20260620 18:36:06.864390 27277 log.cc:1196] T 650dc9a697b44fdfa565b6dbbbceff56 P 06c0f151156347b987a1d38c50a0ee93: Deleting WAL directory at /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980527431848-22676-0/minicluster-data/ts-1/wal/wals/650dc9a697b44fdfa565b6dbbbceff56
I20260620 18:36:06.864905 27277 ts_tablet_manager.cc:1939] T 650dc9a697b44fdfa565b6dbbbceff56 P 06c0f151156347b987a1d38c50a0ee93: Deleting consensus metadata
I20260620 18:36:06.866221 26766 catalog_manager.cc:4841] TS 692c9112662e487db0cf9c0cd37aa499 (127.22.37.1:36377): tablet 650dc9a697b44fdfa565b6dbbbceff56 (table test-table [id=4ee7da6950ca499e8a91ef3bfe93dd10]) successfully deleted
I20260620 18:36:06.866552 26766 catalog_manager.cc:4841] TS 06c0f151156347b987a1d38c50a0ee93 (127.22.37.2:35131): tablet 650dc9a697b44fdfa565b6dbbbceff56 (table test-table [id=4ee7da6950ca499e8a91ef3bfe93dd10]) successfully deleted
I20260620 18:36:06.867493 27278 ts_tablet_manager.cc:1918] T 650dc9a697b44fdfa565b6dbbbceff56 P 49b45ea90f354aebb8a8801be1222b03: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:36:06.867686 27278 log.cc:1196] T 650dc9a697b44fdfa565b6dbbbceff56 P 49b45ea90f354aebb8a8801be1222b03: Deleting WAL directory at /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980527431848-22676-0/minicluster-data/ts-2/wal/wals/650dc9a697b44fdfa565b6dbbbceff56
I20260620 18:36:06.868285 27278 ts_tablet_manager.cc:1939] T 650dc9a697b44fdfa565b6dbbbceff56 P 49b45ea90f354aebb8a8801be1222b03: Deleting consensus metadata
Jun 20 18:36:06 dist-test-slave-5lgp krb5kdc[26735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980564, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.37.1@KRBTEST.COM
I20260620 18:36:06.869917 26763 catalog_manager.cc:4841] TS 49b45ea90f354aebb8a8801be1222b03 (127.22.37.3:37705): tablet 650dc9a697b44fdfa565b6dbbbceff56 (table test-table [id=4ee7da6950ca499e8a91ef3bfe93dd10]) successfully deleted
I20260620 18:36:06.876965 27249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:06.838335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38082 (local address 127.22.37.1:36377)
0620 18:36:06.838507 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:06.838513 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:06.853119 (+ 14606us) server_negotiation.cc:407] Connection header received
0620 18:36:06.853317 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:06.853323 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:06.853438 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:06.853557 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:06.855721 (+ 2164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:06.856862 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:06.858702 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:06.859395 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:06.859489 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:06.870594 (+ 11105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:06.870608 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:06.870625 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:06.870675 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:06.874255 (+ 3580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:06.874919 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:06.874931 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:06.874935 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:06.875093 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:06.875451 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:06.875456 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:06.875459 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:06.875897 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:06.876184 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:06.876374 (+ 190us) server_negotiation.cc:299] Negotiation successful
0620 18:36:06.876584 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 18:36:06.877640 26962 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:36:06.966037 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 26840
I20260620 18:36:06.990393 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 26973
Jun 20 18:36:06 dist-test-slave-5lgp krb5kdc[26735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980564, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.37.2@KRBTEST.COM
I20260620 18:36:07.007215 27245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:06.991251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32966 (local address 127.22.37.2:35131)
0620 18:36:06.991456 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:06.991462 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:06.991484 (+ 22us) server_negotiation.cc:407] Connection header received
0620 18:36:06.991674 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:06.991678 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:06.991759 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:06.991891 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:06.993753 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:06.994483 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:06.996284 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:06.996671 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:06.996771 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:07.001538 (+ 4767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:07.001554 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:07.001572 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:07.001605 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:07.004377 (+ 2772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:07.005100 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:07.005107 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:07.005110 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:07.005234 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:07.005649 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:07.005655 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:07.005658 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:07.006023 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:07.006337 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:07.006632 (+ 295us) server_negotiation.cc:299] Negotiation successful
0620 18:36:07.006853 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 18:36:07.007915 27095 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:36:07.096916 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 26973
I20260620 18:36:07.118954 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 27105
Jun 20 18:36:07 dist-test-slave-5lgp krb5kdc[26735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980564, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.37.3@KRBTEST.COM
I20260620 18:36:07.135349 27263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:07.119734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58036 (local address 127.22.37.3:37705)
0620 18:36:07.119977 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:07.119985 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:07.120006 (+ 21us) server_negotiation.cc:407] Connection header received
0620 18:36:07.120141 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:07.120150 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:07.120214 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:07.120341 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:07.121637 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:07.122399 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:07.123671 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:07.124119 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:07.124273 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:07.129340 (+ 5067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:07.129352 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:07.129419 (+ 67us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:07.129466 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:07.132416 (+ 2950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:07.133236 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:07.133243 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:07.133245 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:07.133448 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:07.133875 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:07.133878 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:07.133880 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:07.134229 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:07.134477 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:07.134852 (+ 375us) server_negotiation.cc:299] Negotiation successful
0620 18:36:07.135034 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260620 18:36:07.136052 27227 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:36:07.262686 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 27105
I20260620 18:36:07.281977 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 26748
I20260620 18:36:07.283102 26809 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:36:07.388634 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 26748
2026-06-20T18:36:07Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4580 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-20T18:36:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:36:07Z Disabled control of system clock
I20260620 18:36:07.455001 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980527431848-22676-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980527431848-22676-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980527431848-22676-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.62:43163
--webserver_interface=127.22.37.62
--webserver_port=0
--builtin_ntp_servers=127.22.37.20:33695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.37.62:43163
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 18:36:07.779421 27288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:36:07.779726 27288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:36:07.779816 27288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:36:07.787031 27288 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:36:07.787155 27288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:36:07.787200 27288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:36:07.787266 27288 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:36:07.787318 27288 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:36:07.796469 27288 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:33695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980527431848-22676-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980527431848-22676-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.37.62:43163
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.62:43163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.37.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980527431848-22676-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:36:07.797881 27288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:36:07.799685 27288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:36:07.812856 27293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:07.812856 27294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:07.812911 27296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:07.814214 27288 server_base.cc:1034] running on GCE node
I20260620 18:36:07.815174 27288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:36:07.817061 27288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:36:07.818318 27288 hybrid_clock.cc:648] HybridClock initialized: now 1781980567818169 us; error 117 us; skew 500 ppm
I20260620 18:36:07.818764 27288 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:36:07.821818 27288 webserver.cc:460] Webserver started at http://127.22.37.62:37927/ using document root <none> and password file <none>
I20260620 18:36:07.822579 27288 fs_manager.cc:359] Metadata directory not provided
I20260620 18:36:07.822701 27288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:36:07.823086 27288 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:36:07.827261 27288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980527431848-22676-0/minicluster-data/master-0/data/instance:
uuid: "18b2ba8695c64e0c93f088a3c73297a8"
format_stamp: "Formatted at 2026-06-20 18:36:07 on dist-test-slave-5lgp"
server_key: "85aa8c5609e9b92a8e9365a1f7b730c5"
server_key_iv: "2eeb08eee4d0c522f617f43198101e0f"
server_key_version: "encryptionkey@0"
I20260620 18:36:07.828409 27288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980527431848-22676-0/minicluster-data/master-0/wal/instance:
uuid: "18b2ba8695c64e0c93f088a3c73297a8"
format_stamp: "Formatted at 2026-06-20 18:36:07 on dist-test-slave-5lgp"
server_key: "85aa8c5609e9b92a8e9365a1f7b730c5"
server_key_iv: "2eeb08eee4d0c522f617f43198101e0f"
server_key_version: "encryptionkey@0"
I20260620 18:36:07.835523 27288 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.003s
I20260620 18:36:07.839870 27302 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:07.841672 27288 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:36:07.841858 27288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980527431848-22676-0/minicluster-data/master-0/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980527431848-22676-0/minicluster-data/master-0/wal
uuid: "18b2ba8695c64e0c93f088a3c73297a8"
format_stamp: "Formatted at 2026-06-20 18:36:07 on dist-test-slave-5lgp"
server_key: "85aa8c5609e9b92a8e9365a1f7b730c5"
server_key_iv: "2eeb08eee4d0c522f617f43198101e0f"
server_key_version: "encryptionkey@0"
I20260620 18:36:07.842098 27288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980527431848-22676-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980527431848-22676-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980527431848-22676-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:36:07.848428 27288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:36:07.849174 27288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:36:07.849522 27288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:36:07.868125 27288 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.62:43163
I20260620 18:36:07.868160 27353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.62:43163 every 8 connection(s)
I20260620 18:36:07.869520 27288 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
I20260620 18:36:07.873137 27354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:36:07.873554 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 27288
I20260620 18:36:07.873751 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980527431848-22676-0/minicluster-data/master-0/wal/instance
I20260620 18:36:07.874379 22676 external_mini_cluster.cc:1428] Setting key af80a67c23c39300a4b94f8bdd9d1aef
I20260620 18:36:07.882392 27357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:07.876325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53384 (local address 127.22.37.62:43163)
0620 18:36:07.877349 (+ 1024us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:07.877360 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:07.877459 (+ 99us) server_negotiation.cc:407] Connection header received
0620 18:36:07.878949 (+ 1490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:07.878969 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:07.879380 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:07.879690 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:07.880825 (+ 1135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:07.880831 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:07.880843 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:36:07.880857 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:07.880916 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:36:07.880930 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:07.881089 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:07.881191 (+ 102us) server_negotiation.cc:299] Negotiation successful
0620 18:36:07.881482 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":398,"threads_started":1}
I20260620 18:36:07.885563 27354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18b2ba8695c64e0c93f088a3c73297a8: Bootstrap starting.
I20260620 18:36:07.888291 27354 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 18b2ba8695c64e0c93f088a3c73297a8: Neither blocks nor log segments found. Creating new log.
I20260620 18:36:07.889233 27354 log.cc:824] T 00000000000000000000000000000000 P 18b2ba8695c64e0c93f088a3c73297a8: Log is configured to *not* fsync() on all Append() calls
I20260620 18:36:07.891799 27354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18b2ba8695c64e0c93f088a3c73297a8: No bootstrap required, opened a new log
I20260620 18:36:07.898622 27354 raft_consensus.cc:348] T 00000000000000000000000000000000 P 18b2ba8695c64e0c93f088a3c73297a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18b2ba8695c64e0c93f088a3c73297a8" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 43163 } }
I20260620 18:36:07.899026 27354 raft_consensus.cc:374] T 00000000000000000000000000000000 P 18b2ba8695c64e0c93f088a3c73297a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:36:07.899152 27354 raft_consensus.cc:729] T 00000000000000000000000000000000 P 18b2ba8695c64e0c93f088a3c73297a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18b2ba8695c64e0c93f088a3c73297a8, State: Initialized, Role: FOLLOWER
I20260620 18:36:07.899659 27354 consensus_queue.cc:260] T 00000000000000000000000000000000 P 18b2ba8695c64e0c93f088a3c73297a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18b2ba8695c64e0c93f088a3c73297a8" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 43163 } }
I20260620 18:36:07.899884 27354 raft_consensus.cc:388] T 00000000000000000000000000000000 P 18b2ba8695c64e0c93f088a3c73297a8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:36:07.900022 27354 raft_consensus.cc:482] T 00000000000000000000000000000000 P 18b2ba8695c64e0c93f088a3c73297a8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:36:07.900199 27354 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 18b2ba8695c64e0c93f088a3c73297a8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:36:07.902233 27354 raft_consensus.cc:504] T 00000000000000000000000000000000 P 18b2ba8695c64e0c93f088a3c73297a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18b2ba8695c64e0c93f088a3c73297a8" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 43163 } }
I20260620 18:36:07.902683 27354 leader_election.cc:302] T 00000000000000000000000000000000 P 18b2ba8695c64e0c93f088a3c73297a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 18b2ba8695c64e0c93f088a3c73297a8; no voters:
I20260620 18:36:07.903216 27354 leader_election.cc:288] T 00000000000000000000000000000000 P 18b2ba8695c64e0c93f088a3c73297a8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:36:07.903313 27359 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 18b2ba8695c64e0c93f088a3c73297a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:36:07.904024 27359 raft_consensus.cc:686] T 00000000000000000000000000000000 P 18b2ba8695c64e0c93f088a3c73297a8 [term 1 LEADER]: Becoming Leader. State: Replica: 18b2ba8695c64e0c93f088a3c73297a8, State: Running, Role: LEADER
I20260620 18:36:07.904495 27359 consensus_queue.cc:237] T 00000000000000000000000000000000 P 18b2ba8695c64e0c93f088a3c73297a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18b2ba8695c64e0c93f088a3c73297a8" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 43163 } }
I20260620 18:36:07.905143 27354 sys_catalog.cc:564] T 00000000000000000000000000000000 P 18b2ba8695c64e0c93f088a3c73297a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:36:07.907219 27361 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18b2ba8695c64e0c93f088a3c73297a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 18b2ba8695c64e0c93f088a3c73297a8. Latest consensus state: current_term: 1 leader_uuid: "18b2ba8695c64e0c93f088a3c73297a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18b2ba8695c64e0c93f088a3c73297a8" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 43163 } } }
I20260620 18:36:07.907632 27361 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18b2ba8695c64e0c93f088a3c73297a8 [sys.catalog]: This master's current role is: LEADER
I20260620 18:36:07.907626 27360 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18b2ba8695c64e0c93f088a3c73297a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "18b2ba8695c64e0c93f088a3c73297a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18b2ba8695c64e0c93f088a3c73297a8" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 43163 } } }
I20260620 18:36:07.908846 27360 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18b2ba8695c64e0c93f088a3c73297a8 [sys.catalog]: This master's current role is: LEADER
I20260620 18:36:07.909837 27366 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:36:07.914644 27366 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:36:07.923156 27366 catalog_manager.cc:1347] Generated new cluster ID: 0b052f9a908e4754ab83f27805d0aa49
I20260620 18:36:07.923271 27366 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:36:07.965234 27366 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:36:07.965700 27366 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:36:07.976917 27366 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 18b2ba8695c64e0c93f088a3c73297a8: Generated new TSK 0
I20260620 18:36:07.977953 27366 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:36:07.995309 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980527431848-22676-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980527431848-22676-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980527431848-22676-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980527431848-22676-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.37.1:0
--local_ip_for_outbound_sockets=127.22.37.1
--webserver_interface=127.22.37.1
--webserver_port=0
--tserver_master_addrs=127.22.37.62:43163
--builtin_ntp_servers=127.22.37.20:33695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 18:36:08.319211 27378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:36:08.319492 27378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:36:08.319608 27378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:36:08.326987 27378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:36:08.327105 27378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:36:08.327291 27378 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.37.1
I20260620 18:36:08.336300 27378 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:33695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980527431848-22676-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980527431848-22676-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.37.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980527431848-22676-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.37.1
--webserver_port=0
--tserver_master_addrs=127.22.37.62:43163
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.37.1
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980527431848-22676-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:36:08.337574 27378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:36:08.339454 27378 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:36:08.352396 27383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:08.352437 27384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:08.352486 27386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:08.358870 27378 server_base.cc:1034] running on GCE node
I20260620 18:36:08.359419 27378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:36:08.360271 27378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:36:08.361490 27378 hybrid_clock.cc:648] HybridClock initialized: now 1781980568361450 us; error 50 us; skew 500 ppm
I20260620 18:36:08.361804 27378 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:36:08.363947 27378 webserver.cc:460] Webserver started at http://127.22.37.1:42329/ using document root <none> and password file <none>
I20260620 18:36:08.364615 27378 fs_manager.cc:359] Metadata directory not provided
I20260620 18:36:08.364759 27378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:36:08.365072 27378 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:36:08.367854 27378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980527431848-22676-0/minicluster-data/ts-0/data/instance:
uuid: "0f43868fca7f4100a11cb2ba2b1075ae"
format_stamp: "Formatted at 2026-06-20 18:36:08 on dist-test-slave-5lgp"
server_key: "2d30b8d61f9014d74d78c75cfdcef315"
server_key_iv: "9b7cda124196bbddec178991a4b1788f"
server_key_version: "encryptionkey@0"
I20260620 18:36:08.368634 27378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980527431848-22676-0/minicluster-data/ts-0/wal/instance:
uuid: "0f43868fca7f4100a11cb2ba2b1075ae"
format_stamp: "Formatted at 2026-06-20 18:36:08 on dist-test-slave-5lgp"
server_key: "2d30b8d61f9014d74d78c75cfdcef315"
server_key_iv: "9b7cda124196bbddec178991a4b1788f"
server_key_version: "encryptionkey@0"
I20260620 18:36:08.373533 27378 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 18:36:08.376725 27392 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:08.378022 27378 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:36:08.378185 27378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980527431848-22676-0/minicluster-data/ts-0/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980527431848-22676-0/minicluster-data/ts-0/wal
uuid: "0f43868fca7f4100a11cb2ba2b1075ae"
format_stamp: "Formatted at 2026-06-20 18:36:08 on dist-test-slave-5lgp"
server_key: "2d30b8d61f9014d74d78c75cfdcef315"
server_key_iv: "9b7cda124196bbddec178991a4b1788f"
server_key_version: "encryptionkey@0"
I20260620 18:36:08.378389 27378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980527431848-22676-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980527431848-22676-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980527431848-22676-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:36:08.383699 27378 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:36:08.384461 27378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:36:08.384706 27378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:36:08.385604 27378 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:36:08.386886 27378 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:36:08.387005 27378 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:08.387130 27378 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:36:08.387215 27378 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:08.419402 27378 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.1:41215
I20260620 18:36:08.419492 27504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.1:41215 every 8 connection(s)
I20260620 18:36:08.420733 27378 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980527431848-22676-0/minicluster-data/ts-0/data/info.pb
I20260620 18:36:08.424961 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 27378
I20260620 18:36:08.425151 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980527431848-22676-0/minicluster-data/ts-0/wal/instance
I20260620 18:36:08.425992 22676 external_mini_cluster.cc:1428] Setting key 071a92fc35ba3efd6752ed76d7e4d93f
I20260620 18:36:08.429184 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980527431848-22676-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980527431848-22676-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980527431848-22676-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980527431848-22676-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.37.2:0
--local_ip_for_outbound_sockets=127.22.37.2
--webserver_interface=127.22.37.2
--webserver_port=0
--tserver_master_addrs=127.22.37.62:43163
--builtin_ntp_servers=127.22.37.20:33695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 18:36:08.429747 27507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:08.423254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.1:44247 (local address 127.22.37.62:43163)
0620 18:36:08.423679 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:08.423683 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:08.424955 (+ 1272us) server_negotiation.cc:407] Connection header received
0620 18:36:08.427310 (+ 2355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:08.427316 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:08.427388 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:08.427512 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:08.428970 (+ 1458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:08.428975 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:08.428978 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:36:08.428994 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:08.429033 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:36:08.429045 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:08.429146 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:08.429433 (+ 287us) server_negotiation.cc:299] Negotiation successful
0620 18:36:08.429497 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":238,"threads_started":1}
I20260620 18:36:08.431059 27508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:08.423735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:43163 (local address 127.22.37.1:44247)
0620 18:36:08.424749 (+ 1014us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:08.424805 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:08.427016 (+ 2211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:08.427772 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:08.427781 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:08.427891 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:08.428608 (+ 717us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:36:08.428611 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:08.428662 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:36:08.428782 (+ 120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:36:08.429151 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:36:08.429161 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:36:08.429171 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 18:36:08.429487 (+ 316us) client_negotiation.cc:240] Negotiation successful
0620 18:36:08.429756 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":271,"threads_started":1}
I20260620 18:36:08.432570 27505 heartbeater.cc:347] Connected to a master server at 127.22.37.62:43163
I20260620 18:36:08.432891 27505 heartbeater.cc:464] Registering TS with master...
I20260620 18:36:08.433595 27505 heartbeater.cc:511] Master 127.22.37.62:43163 requested a full tablet report, sending...
I20260620 18:36:08.435887 27319 ts_manager.cc:194] Registered new tserver with Master: 0f43868fca7f4100a11cb2ba2b1075ae (127.22.37.1:41215)
W20260620 18:36:08.741679 27509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:36:08.741933 27509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:36:08.742017 27509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:36:08.748950 27509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:36:08.749035 27509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:36:08.749133 27509 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.37.2
I20260620 18:36:08.757553 27509 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:33695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980527431848-22676-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980527431848-22676-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.37.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980527431848-22676-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.37.2
--webserver_port=0
--tserver_master_addrs=127.22.37.62:43163
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.37.2
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980527431848-22676-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:36:08.758653 27509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:36:08.760277 27509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:36:08.772686 27514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:08.772720 27517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:08.772776 27515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:08.773793 27509 server_base.cc:1034] running on GCE node
I20260620 18:36:08.774349 27509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:36:08.775205 27509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:36:08.776424 27509 hybrid_clock.cc:648] HybridClock initialized: now 1781980568776368 us; error 45 us; skew 500 ppm
I20260620 18:36:08.776726 27509 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:36:08.778986 27509 webserver.cc:460] Webserver started at http://127.22.37.2:38563/ using document root <none> and password file <none>
I20260620 18:36:08.779661 27509 fs_manager.cc:359] Metadata directory not provided
I20260620 18:36:08.779788 27509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:36:08.780099 27509 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:36:08.783023 27509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980527431848-22676-0/minicluster-data/ts-1/data/instance:
uuid: "4b1d802505024d1db9f82beb783fa681"
format_stamp: "Formatted at 2026-06-20 18:36:08 on dist-test-slave-5lgp"
server_key: "274890479312155166ea2dd5bf988cb3"
server_key_iv: "8b3af77ac2cf0c5b2a03674694e72f9e"
server_key_version: "encryptionkey@0"
I20260620 18:36:08.783797 27509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980527431848-22676-0/minicluster-data/ts-1/wal/instance:
uuid: "4b1d802505024d1db9f82beb783fa681"
format_stamp: "Formatted at 2026-06-20 18:36:08 on dist-test-slave-5lgp"
server_key: "274890479312155166ea2dd5bf988cb3"
server_key_iv: "8b3af77ac2cf0c5b2a03674694e72f9e"
server_key_version: "encryptionkey@0"
I20260620 18:36:08.788760 27509 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 18:36:08.792008 27523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:08.793234 27509 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 18:36:08.793459 27509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980527431848-22676-0/minicluster-data/ts-1/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980527431848-22676-0/minicluster-data/ts-1/wal
uuid: "4b1d802505024d1db9f82beb783fa681"
format_stamp: "Formatted at 2026-06-20 18:36:08 on dist-test-slave-5lgp"
server_key: "274890479312155166ea2dd5bf988cb3"
server_key_iv: "8b3af77ac2cf0c5b2a03674694e72f9e"
server_key_version: "encryptionkey@0"
I20260620 18:36:08.793666 27509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980527431848-22676-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980527431848-22676-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980527431848-22676-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:36:08.799000 27509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:36:08.799803 27509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:36:08.800086 27509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:36:08.800949 27509 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:36:08.802266 27509 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:36:08.802384 27509 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:08.802508 27509 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:36:08.802593 27509 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:08.840483 27509 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.2:35113
I20260620 18:36:08.840528 27635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.2:35113 every 8 connection(s)
I20260620 18:36:08.841897 27509 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980527431848-22676-0/minicluster-data/ts-1/data/info.pb
I20260620 18:36:08.847923 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 27509
I20260620 18:36:08.848095 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980527431848-22676-0/minicluster-data/ts-1/wal/instance
I20260620 18:36:08.848532 22676 external_mini_cluster.cc:1428] Setting key 0d62ba6db9383f7b4cc007ff95b2a699
I20260620 18:36:08.850183 27507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:08.844427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.2:40503 (local address 127.22.37.62:43163)
0620 18:36:08.844653 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:08.844657 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:08.846099 (+ 1442us) server_negotiation.cc:407] Connection header received
0620 18:36:08.847797 (+ 1698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:08.847803 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:08.847877 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:08.847987 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:08.849496 (+ 1509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:08.849502 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:08.849506 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:36:08.849526 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:08.849565 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:36:08.849580 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:08.849705 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:08.849919 (+ 214us) server_negotiation.cc:299] Negotiation successful
0620 18:36:08.849967 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 18:36:08.851337 27638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:08.844947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:43163 (local address 127.22.37.2:40503)
0620 18:36:08.845937 (+ 990us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:08.845979 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:08.847511 (+ 1532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:08.848271 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:08.848280 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:08.848377 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:08.849081 (+ 704us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:36:08.849083 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:08.849144 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:36:08.849271 (+ 127us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:36:08.849697 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:36:08.849706 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:36:08.849713 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 18:36:08.850003 (+ 290us) client_negotiation.cc:240] Negotiation successful
0620 18:36:08.850280 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":295,"threads_started":1}
I20260620 18:36:08.851644 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980527431848-22676-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980527431848-22676-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980527431848-22676-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980527431848-22676-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.37.3:0
--local_ip_for_outbound_sockets=127.22.37.3
--webserver_interface=127.22.37.3
--webserver_port=0
--tserver_master_addrs=127.22.37.62:43163
--builtin_ntp_servers=127.22.37.20:33695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 18:36:08.852664 27636 heartbeater.cc:347] Connected to a master server at 127.22.37.62:43163
I20260620 18:36:08.852985 27636 heartbeater.cc:464] Registering TS with master...
I20260620 18:36:08.853714 27636 heartbeater.cc:511] Master 127.22.37.62:43163 requested a full tablet report, sending...
I20260620 18:36:08.855317 27319 ts_manager.cc:194] Registered new tserver with Master: 4b1d802505024d1db9f82beb783fa681 (127.22.37.2:35113)
W20260620 18:36:09.188925 27639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:36:09.189258 27639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:36:09.189478 27639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:36:09.196798 27639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:36:09.196936 27639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:36:09.197119 27639 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.37.3
I20260620 18:36:09.206598 27639 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:33695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980527431848-22676-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980527431848-22676-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.37.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980527431848-22676-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.37.3
--webserver_port=0
--tserver_master_addrs=127.22.37.62:43163
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.37.3
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980527431848-22676-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:36:09.207940 27639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:36:09.209890 27639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:36:09.222959 27647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:09.222913 27645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:09.223412 27644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:09.223667 27639 server_base.cc:1034] running on GCE node
I20260620 18:36:09.224300 27639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:36:09.225281 27639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:36:09.226565 27639 hybrid_clock.cc:648] HybridClock initialized: now 1781980569226520 us; error 84 us; skew 500 ppm
I20260620 18:36:09.226902 27639 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:36:09.229027 27639 webserver.cc:460] Webserver started at http://127.22.37.3:39085/ using document root <none> and password file <none>
I20260620 18:36:09.229777 27639 fs_manager.cc:359] Metadata directory not provided
I20260620 18:36:09.229911 27639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:36:09.230234 27639 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:36:09.233075 27639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980527431848-22676-0/minicluster-data/ts-2/data/instance:
uuid: "0b6e56e8210041b1ad2ec22acf97474c"
format_stamp: "Formatted at 2026-06-20 18:36:09 on dist-test-slave-5lgp"
server_key: "657a88e6fd48ce5f67ec167254eb7ef0"
server_key_iv: "9109f1a20ad1c13ba5ef1acda6993696"
server_key_version: "encryptionkey@0"
I20260620 18:36:09.233930 27639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980527431848-22676-0/minicluster-data/ts-2/wal/instance:
uuid: "0b6e56e8210041b1ad2ec22acf97474c"
format_stamp: "Formatted at 2026-06-20 18:36:09 on dist-test-slave-5lgp"
server_key: "657a88e6fd48ce5f67ec167254eb7ef0"
server_key_iv: "9109f1a20ad1c13ba5ef1acda6993696"
server_key_version: "encryptionkey@0"
I20260620 18:36:09.238790 27639 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 18:36:09.242033 27653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:09.243248 27639 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:36:09.243436 27639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980527431848-22676-0/minicluster-data/ts-2/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980527431848-22676-0/minicluster-data/ts-2/wal
uuid: "0b6e56e8210041b1ad2ec22acf97474c"
format_stamp: "Formatted at 2026-06-20 18:36:09 on dist-test-slave-5lgp"
server_key: "657a88e6fd48ce5f67ec167254eb7ef0"
server_key_iv: "9109f1a20ad1c13ba5ef1acda6993696"
server_key_version: "encryptionkey@0"
I20260620 18:36:09.243681 27639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980527431848-22676-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980527431848-22676-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980527431848-22676-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:36:09.248886 27639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:36:09.249666 27639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:36:09.249946 27639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:36:09.250787 27639 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:36:09.252064 27639 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:36:09.252187 27639 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:09.252332 27639 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:36:09.252420 27639 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:09.285094 27639 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.3:44303
I20260620 18:36:09.285142 27765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.3:44303 every 8 connection(s)
I20260620 18:36:09.286504 27639 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980527431848-22676-0/minicluster-data/ts-2/data/info.pb
I20260620 18:36:09.291352 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 27639
I20260620 18:36:09.291533 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980527431848-22676-0/minicluster-data/ts-2/wal/instance
I20260620 18:36:09.292106 22676 external_mini_cluster.cc:1428] Setting key 4f50a2ccd762e4754dc63c587ec154da
I20260620 18:36:09.295835 27507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:09.289285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.3:59377 (local address 127.22.37.62:43163)
0620 18:36:09.289537 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:09.289544 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:09.291194 (+ 1650us) server_negotiation.cc:407] Connection header received
0620 18:36:09.293440 (+ 2246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:09.293447 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:09.293522 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:09.293641 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:09.295135 (+ 1494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:09.295140 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:09.295145 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:36:09.295164 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:09.295201 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:36:09.295218 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:09.295341 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:09.295581 (+ 240us) server_negotiation.cc:299] Negotiation successful
0620 18:36:09.295629 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260620 18:36:09.296996 27768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:09.289973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:43163 (local address 127.22.37.3:59377)
0620 18:36:09.290978 (+ 1005us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:09.291038 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:09.293084 (+ 2046us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:09.293900 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:09.293909 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:09.294010 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:09.294790 (+ 780us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:36:09.294793 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:09.294834 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:36:09.294921 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:36:09.295354 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:36:09.295364 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:36:09.295372 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 18:36:09.295642 (+ 270us) client_negotiation.cc:240] Negotiation successful
0620 18:36:09.295859 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":315,"threads_started":1}
I20260620 18:36:09.298461 27766 heartbeater.cc:347] Connected to a master server at 127.22.37.62:43163
I20260620 18:36:09.298714 27766 heartbeater.cc:464] Registering TS with master...
I20260620 18:36:09.299232 27766 heartbeater.cc:511] Master 127.22.37.62:43163 requested a full tablet report, sending...
I20260620 18:36:09.300593 27319 ts_manager.cc:194] Registered new tserver with Master: 0b6e56e8210041b1ad2ec22acf97474c (127.22.37.3:44303)
I20260620 18:36:09.307646 22676 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:36:09.314761 27507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:09.312132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39542 (local address 127.22.37.62:43163)
0620 18:36:09.312300 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:09.312305 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:09.312851 (+ 546us) server_negotiation.cc:407] Connection header received
0620 18:36:09.313029 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:09.313033 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:09.313086 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:09.313188 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:09.314129 (+ 941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:09.314135 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:09.314140 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:36:09.314159 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:09.314193 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:36:09.314204 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:09.314328 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:09.314438 (+ 110us) server_negotiation.cc:299] Negotiation successful
0620 18:36:09.314495 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 18:36:09.318408 27319 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39542:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:36:09.324944 27319 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:36:09.355518 27779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:09.336720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42938 (local address 127.22.37.1:41215)
0620 18:36:09.352372 (+ 15652us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:09.352382 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:09.352417 (+ 35us) server_negotiation.cc:407] Connection header received
0620 18:36:09.352468 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:09.352499 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:09.352750 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:09.352909 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:09.354607 (+ 1698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:09.354619 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:09.354635 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:36:09.354653 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:09.354708 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:36:09.354730 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:09.354859 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:09.355046 (+ 187us) server_negotiation.cc:299] Negotiation successful
0620 18:36:09.355179 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15508,"thread_start_us":15390,"threads_started":1}
I20260620 18:36:09.355556 27780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:09.336571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.1:41215 (local address 127.0.0.1:42938)
0620 18:36:09.350869 (+ 14298us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:09.350914 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:09.351059 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:09.352937 (+ 1878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:09.352945 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:09.353035 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:09.354116 (+ 1081us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:36:09.354121 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:09.354154 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:36:09.354242 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:36:09.354816 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:36:09.354826 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:36:09.354833 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 18:36:09.355045 (+ 212us) client_negotiation.cc:240] Negotiation successful
0620 18:36:09.355152 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14022,"thread_start_us":15250,"threads_started":1}
I20260620 18:36:09.355912 27782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:09.337548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57878 (local address 127.22.37.2:35113)
0620 18:36:09.349962 (+ 12414us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:09.349978 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:09.351730 (+ 1752us) server_negotiation.cc:407] Connection header received
0620 18:36:09.352025 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:09.352035 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:09.352341 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:09.352486 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:09.354862 (+ 2376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:09.354871 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:09.354882 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:36:09.354912 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:09.354973 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:36:09.354999 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:09.355132 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:09.355419 (+ 287us) server_negotiation.cc:299] Negotiation successful
0620 18:36:09.355538 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12161,"thread_start_us":12453,"threads_started":1}
I20260620 18:36:09.356319 27778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:09.337276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.3:44303 (local address 127.0.0.1:41316)
0620 18:36:09.352409 (+ 15133us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:09.352431 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:09.352532 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:09.353544 (+ 1012us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:09.353549 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:09.353572 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:09.354453 (+ 881us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:36:09.354457 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:09.354472 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:36:09.354514 (+ 42us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:36:09.355617 (+ 1103us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:36:09.355622 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:36:09.355628 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 18:36:09.355772 (+ 144us) client_negotiation.cc:240] Negotiation successful
0620 18:36:09.355824 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14698,"thread_start_us":14015,"threads_started":1}
I20260620 18:36:09.356822 27781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:09.336821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.2:35113 (local address 127.0.0.1:57878)
0620 18:36:09.351593 (+ 14772us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:09.351611 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:09.351835 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:09.353584 (+ 1749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:09.353592 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:09.353623 (+ 31us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:09.354594 (+ 971us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:36:09.354597 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:09.354614 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:36:09.354662 (+ 48us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:36:09.355238 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:36:09.355243 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:36:09.355248 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 18:36:09.356327 (+ 1079us) client_negotiation.cc:240] Negotiation successful
0620 18:36:09.356371 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14628,"thread_start_us":13610,"threads_started":1}
I20260620 18:36:09.356904 27783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:09.337745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41316 (local address 127.22.37.3:44303)
0620 18:36:09.351662 (+ 13917us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:09.351675 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:09.352534 (+ 859us) server_negotiation.cc:407] Connection header received
0620 18:36:09.352706 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:09.352715 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:09.352925 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:09.353057 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:09.355259 (+ 2202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:09.355267 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:09.355281 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:36:09.355304 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:09.355369 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:36:09.355390 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:09.355558 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:09.356342 (+ 784us) server_negotiation.cc:299] Negotiation successful
0620 18:36:09.356443 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13735,"thread_start_us":13560,"threads_started":1}
I20260620 18:36:09.359190 27440 tablet_service.cc:1467] Processing CreateTablet for tablet ad82d8f5869b4eeb893181081f1c55e7 (DEFAULT_TABLE table=test-table [id=5bd9822ef57e4b55820abd6fdf9e570f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:36:09.360540 27440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad82d8f5869b4eeb893181081f1c55e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:36:09.360278 27571 tablet_service.cc:1467] Processing CreateTablet for tablet ad82d8f5869b4eeb893181081f1c55e7 (DEFAULT_TABLE table=test-table [id=5bd9822ef57e4b55820abd6fdf9e570f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:36:09.360278 27701 tablet_service.cc:1467] Processing CreateTablet for tablet ad82d8f5869b4eeb893181081f1c55e7 (DEFAULT_TABLE table=test-table [id=5bd9822ef57e4b55820abd6fdf9e570f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:36:09.361660 27571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad82d8f5869b4eeb893181081f1c55e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:36:09.361660 27701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad82d8f5869b4eeb893181081f1c55e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:36:09.371250 27784 tablet_bootstrap.cc:492] T ad82d8f5869b4eeb893181081f1c55e7 P 0f43868fca7f4100a11cb2ba2b1075ae: Bootstrap starting.
I20260620 18:36:09.374248 27784 tablet_bootstrap.cc:654] T ad82d8f5869b4eeb893181081f1c55e7 P 0f43868fca7f4100a11cb2ba2b1075ae: Neither blocks nor log segments found. Creating new log.
I20260620 18:36:09.375208 27784 log.cc:824] T ad82d8f5869b4eeb893181081f1c55e7 P 0f43868fca7f4100a11cb2ba2b1075ae: Log is configured to *not* fsync() on all Append() calls
I20260620 18:36:09.378041 27787 tablet_bootstrap.cc:492] T ad82d8f5869b4eeb893181081f1c55e7 P 0b6e56e8210041b1ad2ec22acf97474c: Bootstrap starting.
I20260620 18:36:09.379441 27784 tablet_bootstrap.cc:492] T ad82d8f5869b4eeb893181081f1c55e7 P 0f43868fca7f4100a11cb2ba2b1075ae: No bootstrap required, opened a new log
I20260620 18:36:09.379722 27784 ts_tablet_manager.cc:1397] T ad82d8f5869b4eeb893181081f1c55e7 P 0f43868fca7f4100a11cb2ba2b1075ae: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260620 18:36:09.381632 27787 tablet_bootstrap.cc:654] T ad82d8f5869b4eeb893181081f1c55e7 P 0b6e56e8210041b1ad2ec22acf97474c: Neither blocks nor log segments found. Creating new log.
I20260620 18:36:09.382189 27785 tablet_bootstrap.cc:492] T ad82d8f5869b4eeb893181081f1c55e7 P 4b1d802505024d1db9f82beb783fa681: Bootstrap starting.
I20260620 18:36:09.383111 27787 log.cc:824] T ad82d8f5869b4eeb893181081f1c55e7 P 0b6e56e8210041b1ad2ec22acf97474c: Log is configured to *not* fsync() on all Append() calls
I20260620 18:36:09.385910 27785 tablet_bootstrap.cc:654] T ad82d8f5869b4eeb893181081f1c55e7 P 4b1d802505024d1db9f82beb783fa681: Neither blocks nor log segments found. Creating new log.
I20260620 18:36:09.386358 27787 tablet_bootstrap.cc:492] T ad82d8f5869b4eeb893181081f1c55e7 P 0b6e56e8210041b1ad2ec22acf97474c: No bootstrap required, opened a new log
I20260620 18:36:09.386663 27787 ts_tablet_manager.cc:1397] T ad82d8f5869b4eeb893181081f1c55e7 P 0b6e56e8210041b1ad2ec22acf97474c: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260620 18:36:09.387192 27785 log.cc:824] T ad82d8f5869b4eeb893181081f1c55e7 P 4b1d802505024d1db9f82beb783fa681: Log is configured to *not* fsync() on all Append() calls
I20260620 18:36:09.389936 27785 tablet_bootstrap.cc:492] T ad82d8f5869b4eeb893181081f1c55e7 P 4b1d802505024d1db9f82beb783fa681: No bootstrap required, opened a new log
I20260620 18:36:09.389724 27784 raft_consensus.cc:348] T ad82d8f5869b4eeb893181081f1c55e7 P 0f43868fca7f4100a11cb2ba2b1075ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f43868fca7f4100a11cb2ba2b1075ae" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 41215 } } peers { permanent_uuid: "4b1d802505024d1db9f82beb783fa681" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 35113 } } peers { permanent_uuid: "0b6e56e8210041b1ad2ec22acf97474c" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 44303 } }
I20260620 18:36:09.390154 27784 raft_consensus.cc:374] T ad82d8f5869b4eeb893181081f1c55e7 P 0f43868fca7f4100a11cb2ba2b1075ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:36:09.390223 27785 ts_tablet_manager.cc:1397] T ad82d8f5869b4eeb893181081f1c55e7 P 4b1d802505024d1db9f82beb783fa681: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260620 18:36:09.390251 27784 raft_consensus.cc:729] T ad82d8f5869b4eeb893181081f1c55e7 P 0f43868fca7f4100a11cb2ba2b1075ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f43868fca7f4100a11cb2ba2b1075ae, State: Initialized, Role: FOLLOWER
I20260620 18:36:09.390857 27784 consensus_queue.cc:260] T ad82d8f5869b4eeb893181081f1c55e7 P 0f43868fca7f4100a11cb2ba2b1075ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f43868fca7f4100a11cb2ba2b1075ae" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 41215 } } peers { permanent_uuid: "4b1d802505024d1db9f82beb783fa681" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 35113 } } peers { permanent_uuid: "0b6e56e8210041b1ad2ec22acf97474c" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 44303 } }
I20260620 18:36:09.391992 27505 heartbeater.cc:503] Master 127.22.37.62:43163 was elected leader, sending a full tablet report...
I20260620 18:36:09.392525 27784 ts_tablet_manager.cc:1428] T ad82d8f5869b4eeb893181081f1c55e7 P 0f43868fca7f4100a11cb2ba2b1075ae: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260620 18:36:09.396399 27787 raft_consensus.cc:348] T ad82d8f5869b4eeb893181081f1c55e7 P 0b6e56e8210041b1ad2ec22acf97474c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f43868fca7f4100a11cb2ba2b1075ae" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 41215 } } peers { permanent_uuid: "4b1d802505024d1db9f82beb783fa681" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 35113 } } peers { permanent_uuid: "0b6e56e8210041b1ad2ec22acf97474c" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 44303 } }
I20260620 18:36:09.396731 27787 raft_consensus.cc:374] T ad82d8f5869b4eeb893181081f1c55e7 P 0b6e56e8210041b1ad2ec22acf97474c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:36:09.396842 27787 raft_consensus.cc:729] T ad82d8f5869b4eeb893181081f1c55e7 P 0b6e56e8210041b1ad2ec22acf97474c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6e56e8210041b1ad2ec22acf97474c, State: Initialized, Role: FOLLOWER
I20260620 18:36:09.397511 27787 consensus_queue.cc:260] T ad82d8f5869b4eeb893181081f1c55e7 P 0b6e56e8210041b1ad2ec22acf97474c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f43868fca7f4100a11cb2ba2b1075ae" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 41215 } } peers { permanent_uuid: "4b1d802505024d1db9f82beb783fa681" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 35113 } } peers { permanent_uuid: "0b6e56e8210041b1ad2ec22acf97474c" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 44303 } }
I20260620 18:36:09.398728 27766 heartbeater.cc:503] Master 127.22.37.62:43163 was elected leader, sending a full tablet report...
I20260620 18:36:09.399222 27787 ts_tablet_manager.cc:1428] T ad82d8f5869b4eeb893181081f1c55e7 P 0b6e56e8210041b1ad2ec22acf97474c: Time spent starting tablet: real 0.012s user 0.008s sys 0.004s
I20260620 18:36:09.399636 27785 raft_consensus.cc:348] T ad82d8f5869b4eeb893181081f1c55e7 P 4b1d802505024d1db9f82beb783fa681 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f43868fca7f4100a11cb2ba2b1075ae" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 41215 } } peers { permanent_uuid: "4b1d802505024d1db9f82beb783fa681" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 35113 } } peers { permanent_uuid: "0b6e56e8210041b1ad2ec22acf97474c" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 44303 } }
I20260620 18:36:09.400187 27785 raft_consensus.cc:374] T ad82d8f5869b4eeb893181081f1c55e7 P 4b1d802505024d1db9f82beb783fa681 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:36:09.400303 27785 raft_consensus.cc:729] T ad82d8f5869b4eeb893181081f1c55e7 P 4b1d802505024d1db9f82beb783fa681 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b1d802505024d1db9f82beb783fa681, State: Initialized, Role: FOLLOWER
I20260620 18:36:09.400942 27785 consensus_queue.cc:260] T ad82d8f5869b4eeb893181081f1c55e7 P 4b1d802505024d1db9f82beb783fa681 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f43868fca7f4100a11cb2ba2b1075ae" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 41215 } } peers { permanent_uuid: "4b1d802505024d1db9f82beb783fa681" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 35113 } } peers { permanent_uuid: "0b6e56e8210041b1ad2ec22acf97474c" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 44303 } }
I20260620 18:36:09.401995 27636 heartbeater.cc:503] Master 127.22.37.62:43163 was elected leader, sending a full tablet report...
I20260620 18:36:09.402753 27785 ts_tablet_manager.cc:1428] T ad82d8f5869b4eeb893181081f1c55e7 P 4b1d802505024d1db9f82beb783fa681: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260620 18:36:09.406647 27791 raft_consensus.cc:482] T ad82d8f5869b4eeb893181081f1c55e7 P 0b6e56e8210041b1ad2ec22acf97474c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:36:09.406841 27791 raft_consensus.cc:504] T ad82d8f5869b4eeb893181081f1c55e7 P 0b6e56e8210041b1ad2ec22acf97474c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f43868fca7f4100a11cb2ba2b1075ae" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 41215 } } peers { permanent_uuid: "4b1d802505024d1db9f82beb783fa681" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 35113 } } peers { permanent_uuid: "0b6e56e8210041b1ad2ec22acf97474c" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 44303 } }
I20260620 18:36:09.408291 27791 leader_election.cc:288] T ad82d8f5869b4eeb893181081f1c55e7 P 0b6e56e8210041b1ad2ec22acf97474c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f43868fca7f4100a11cb2ba2b1075ae (127.22.37.1:41215), 4b1d802505024d1db9f82beb783fa681 (127.22.37.2:35113)
I20260620 18:36:09.411358 27768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:09.408598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.2:35113 (local address 127.22.37.3:60813)
0620 18:36:09.408869 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:09.408892 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:09.408965 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:09.409865 (+ 900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:09.409870 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:09.409920 (+ 50us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:09.410585 (+ 665us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:36:09.410587 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:09.410601 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:36:09.410640 (+ 39us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:36:09.410933 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:36:09.410938 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:36:09.410943 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 18:36:09.411063 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 18:36:09.411119 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147}
I20260620 18:36:09.411381 27782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:09.409445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.3:60813 (local address 127.22.37.2:35113)
0620 18:36:09.409636 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:09.409640 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:09.409655 (+ 15us) server_negotiation.cc:407] Connection header received
0620 18:36:09.409695 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:09.409699 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:09.409738 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:09.409839 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:09.410777 (+ 938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:09.410783 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:09.410788 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:36:09.410805 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:09.410840 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:36:09.410855 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:09.410953 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:09.411070 (+ 117us) server_negotiation.cc:299] Negotiation successful
0620 18:36:09.411121 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 18:36:09.411983 27779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:09.408666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.3:56151 (local address 127.22.37.1:41215)
0620 18:36:09.409120 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:09.409127 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:09.409958 (+ 831us) server_negotiation.cc:407] Connection header received
0620 18:36:09.410021 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:09.410028 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:09.410088 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:09.410185 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:09.411353 (+ 1168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:09.411358 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:09.411362 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:36:09.411380 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:09.411432 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:36:09.411449 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:09.411570 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:09.411656 (+ 86us) server_negotiation.cc:299] Negotiation successful
0620 18:36:09.411711 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307}
I20260620 18:36:09.412098 27793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:09.408604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.1:41215 (local address 127.22.37.3:56151)
0620 18:36:09.409346 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:09.409432 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:09.409542 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:09.410411 (+ 869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:09.410417 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:09.410441 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:09.411161 (+ 720us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:36:09.411164 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:09.411176 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:36:09.411208 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:36:09.411542 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:36:09.411549 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:36:09.411557 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 18:36:09.411654 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 18:36:09.411689 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":633,"mutex_wait_us":35,"thread_start_us":846,"threads_started":1}
I20260620 18:36:09.412528 27591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad82d8f5869b4eeb893181081f1c55e7" candidate_uuid: "0b6e56e8210041b1ad2ec22acf97474c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b1d802505024d1db9f82beb783fa681" is_pre_election: true
I20260620 18:36:09.412961 27591 raft_consensus.cc:2454] T ad82d8f5869b4eeb893181081f1c55e7 P 4b1d802505024d1db9f82beb783fa681 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b6e56e8210041b1ad2ec22acf97474c in term 0.
I20260620 18:36:09.413586 27656 leader_election.cc:302] T ad82d8f5869b4eeb893181081f1c55e7 P 0b6e56e8210041b1ad2ec22acf97474c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6e56e8210041b1ad2ec22acf97474c, 4b1d802505024d1db9f82beb783fa681; no voters:
I20260620 18:36:09.413957 27791 raft_consensus.cc:2781] T ad82d8f5869b4eeb893181081f1c55e7 P 0b6e56e8210041b1ad2ec22acf97474c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:36:09.414104 27791 raft_consensus.cc:482] T ad82d8f5869b4eeb893181081f1c55e7 P 0b6e56e8210041b1ad2ec22acf97474c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:36:09.414222 27791 raft_consensus.cc:3037] T ad82d8f5869b4eeb893181081f1c55e7 P 0b6e56e8210041b1ad2ec22acf97474c [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:36:09.414143 27460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad82d8f5869b4eeb893181081f1c55e7" candidate_uuid: "0b6e56e8210041b1ad2ec22acf97474c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f43868fca7f4100a11cb2ba2b1075ae" is_pre_election: true
I20260620 18:36:09.414638 27460 raft_consensus.cc:2454] T ad82d8f5869b4eeb893181081f1c55e7 P 0f43868fca7f4100a11cb2ba2b1075ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b6e56e8210041b1ad2ec22acf97474c in term 0.
I20260620 18:36:09.416956 27791 raft_consensus.cc:504] T ad82d8f5869b4eeb893181081f1c55e7 P 0b6e56e8210041b1ad2ec22acf97474c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f43868fca7f4100a11cb2ba2b1075ae" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 41215 } } peers { permanent_uuid: "4b1d802505024d1db9f82beb783fa681" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 35113 } } peers { permanent_uuid: "0b6e56e8210041b1ad2ec22acf97474c" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 44303 } }
I20260620 18:36:09.417761 27791 leader_election.cc:288] T ad82d8f5869b4eeb893181081f1c55e7 P 0b6e56e8210041b1ad2ec22acf97474c [CANDIDATE]: Term 1 election: Requested vote from peers 0f43868fca7f4100a11cb2ba2b1075ae (127.22.37.1:41215), 4b1d802505024d1db9f82beb783fa681 (127.22.37.2:35113)
I20260620 18:36:09.417987 27460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad82d8f5869b4eeb893181081f1c55e7" candidate_uuid: "0b6e56e8210041b1ad2ec22acf97474c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f43868fca7f4100a11cb2ba2b1075ae"
I20260620 18:36:09.418032 27591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad82d8f5869b4eeb893181081f1c55e7" candidate_uuid: "0b6e56e8210041b1ad2ec22acf97474c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b1d802505024d1db9f82beb783fa681"
I20260620 18:36:09.418147 27460 raft_consensus.cc:3037] T ad82d8f5869b4eeb893181081f1c55e7 P 0f43868fca7f4100a11cb2ba2b1075ae [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:36:09.418277 27591 raft_consensus.cc:3037] T ad82d8f5869b4eeb893181081f1c55e7 P 4b1d802505024d1db9f82beb783fa681 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:36:09.420470 27460 raft_consensus.cc:2454] T ad82d8f5869b4eeb893181081f1c55e7 P 0f43868fca7f4100a11cb2ba2b1075ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b6e56e8210041b1ad2ec22acf97474c in term 1.
I20260620 18:36:09.420852 27657 leader_election.cc:302] T ad82d8f5869b4eeb893181081f1c55e7 P 0b6e56e8210041b1ad2ec22acf97474c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6e56e8210041b1ad2ec22acf97474c, 0f43868fca7f4100a11cb2ba2b1075ae; no voters:
I20260620 18:36:09.421147 27791 raft_consensus.cc:2781] T ad82d8f5869b4eeb893181081f1c55e7 P 0b6e56e8210041b1ad2ec22acf97474c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:36:09.421502 27591 raft_consensus.cc:2454] T ad82d8f5869b4eeb893181081f1c55e7 P 4b1d802505024d1db9f82beb783fa681 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b6e56e8210041b1ad2ec22acf97474c in term 1.
I20260620 18:36:09.421741 27791 raft_consensus.cc:686] T ad82d8f5869b4eeb893181081f1c55e7 P 0b6e56e8210041b1ad2ec22acf97474c [term 1 LEADER]: Becoming Leader. State: Replica: 0b6e56e8210041b1ad2ec22acf97474c, State: Running, Role: LEADER
I20260620 18:36:09.422204 27791 consensus_queue.cc:237] T ad82d8f5869b4eeb893181081f1c55e7 P 0b6e56e8210041b1ad2ec22acf97474c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f43868fca7f4100a11cb2ba2b1075ae" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 41215 } } peers { permanent_uuid: "4b1d802505024d1db9f82beb783fa681" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 35113 } } peers { permanent_uuid: "0b6e56e8210041b1ad2ec22acf97474c" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 44303 } }
W20260620 18:36:09.423115 27506 tablet.cc:2369] T ad82d8f5869b4eeb893181081f1c55e7 P 0f43868fca7f4100a11cb2ba2b1075ae: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:36:09.427487 27318 catalog_manager.cc:5495] T ad82d8f5869b4eeb893181081f1c55e7 P 0b6e56e8210041b1ad2ec22acf97474c reported cstate change: term changed from 0 to 1, leader changed from <none> to 0b6e56e8210041b1ad2ec22acf97474c (127.22.37.3). New cstate: current_term: 1 leader_uuid: "0b6e56e8210041b1ad2ec22acf97474c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f43868fca7f4100a11cb2ba2b1075ae" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 41215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b1d802505024d1db9f82beb783fa681" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 35113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b6e56e8210041b1ad2ec22acf97474c" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 44303 } health_report { overall_health: HEALTHY } } }
I20260620 18:36:09.444231 27783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:09.442427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41320 (local address 127.22.37.3:44303)
0620 18:36:09.442581 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:09.442585 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:09.442616 (+ 31us) server_negotiation.cc:407] Connection header received
0620 18:36:09.442804 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:09.442807 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:09.442883 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:09.442976 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:09.443796 (+ 820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:09.443799 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:09.443802 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:36:09.443815 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:09.443841 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:36:09.443851 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:09.443959 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:09.444025 (+ 66us) server_negotiation.cc:299] Negotiation successful
0620 18:36:09.444065 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 18:36:09.452514 27460 raft_consensus.cc:1261] T ad82d8f5869b4eeb893181081f1c55e7 P 0f43868fca7f4100a11cb2ba2b1075ae [term 1 FOLLOWER]: Refusing update from remote peer 0b6e56e8210041b1ad2ec22acf97474c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:36:09.452514 27591 raft_consensus.cc:1261] T ad82d8f5869b4eeb893181081f1c55e7 P 4b1d802505024d1db9f82beb783fa681 [term 1 FOLLOWER]: Refusing update from remote peer 0b6e56e8210041b1ad2ec22acf97474c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:36:09.453581 27794 consensus_queue.cc:1035] T ad82d8f5869b4eeb893181081f1c55e7 P 0b6e56e8210041b1ad2ec22acf97474c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b1d802505024d1db9f82beb783fa681" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 35113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:36:09.453976 27791 consensus_queue.cc:1035] T ad82d8f5869b4eeb893181081f1c55e7 P 0b6e56e8210041b1ad2ec22acf97474c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f43868fca7f4100a11cb2ba2b1075ae" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 41215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:36:09.464766 27797 mvcc.cc:205] Tried to move back new op lower bound from 7298992412467503104 to 7298992412357394432. Current Snapshot: MvccSnapshot[applied={T|T < 7298992412467503104}]
I20260620 18:36:09.476663 27798 mvcc.cc:205] Tried to move back new op lower bound from 7298992412467503104 to 7298992412357394432. Current Snapshot: MvccSnapshot[applied={T|T < 7298992412467503104}]
I20260620 18:36:09.480291 27319 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39542:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:36:09.480520 27319 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39542:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:36:09.484875 27319 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 18b2ba8695c64e0c93f088a3c73297a8: Sending DeleteTablet for 3 replicas of tablet ad82d8f5869b4eeb893181081f1c55e7
I20260620 18:36:09.486141 27571 tablet_service.cc:1514] Processing DeleteTablet for tablet ad82d8f5869b4eeb893181081f1c55e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:36:09 UTC) from {username='slave'} at 127.0.0.1:57878
I20260620 18:36:09.486292 27440 tablet_service.cc:1514] Processing DeleteTablet for tablet ad82d8f5869b4eeb893181081f1c55e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:36:09 UTC) from {username='slave'} at 127.0.0.1:42938
I20260620 18:36:09.487180 27807 tablet_replica.cc:330] T ad82d8f5869b4eeb893181081f1c55e7 P 4b1d802505024d1db9f82beb783fa681: stopping tablet replica
I20260620 18:36:09.487762 27807 raft_consensus.cc:2229] T ad82d8f5869b4eeb893181081f1c55e7 P 4b1d802505024d1db9f82beb783fa681 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:36:09.487879 27808 tablet_replica.cc:330] T ad82d8f5869b4eeb893181081f1c55e7 P 0f43868fca7f4100a11cb2ba2b1075ae: stopping tablet replica
I20260620 18:36:09.488219 27701 tablet_service.cc:1514] Processing DeleteTablet for tablet ad82d8f5869b4eeb893181081f1c55e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:36:09 UTC) from {username='slave'} at 127.0.0.1:41316
I20260620 18:36:09.488438 27807 raft_consensus.cc:2258] T ad82d8f5869b4eeb893181081f1c55e7 P 4b1d802505024d1db9f82beb783fa681 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:36:09.488575 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 27378
I20260620 18:36:09.488873 27700 tablet_service.cc:1514] Processing DeleteTablet for tablet ad82d8f5869b4eeb893181081f1c55e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:36:09 UTC) from {username='slave'} at 127.0.0.1:41316
I20260620 18:36:09.489267 27809 tablet_replica.cc:330] T ad82d8f5869b4eeb893181081f1c55e7 P 0b6e56e8210041b1ad2ec22acf97474c: stopping tablet replica
I20260620 18:36:09.489567 27808 raft_consensus.cc:2229] T ad82d8f5869b4eeb893181081f1c55e7 P 0f43868fca7f4100a11cb2ba2b1075ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:36:09.489795 27809 raft_consensus.cc:2229] T ad82d8f5869b4eeb893181081f1c55e7 P 0b6e56e8210041b1ad2ec22acf97474c [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:36:09.490070 27808 raft_consensus.cc:2258] T ad82d8f5869b4eeb893181081f1c55e7 P 0f43868fca7f4100a11cb2ba2b1075ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:36:09.490509 27809 raft_consensus.cc:2258] T ad82d8f5869b4eeb893181081f1c55e7 P 0b6e56e8210041b1ad2ec22acf97474c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:36:09.490900 27807 ts_tablet_manager.cc:1905] T ad82d8f5869b4eeb893181081f1c55e7 P 4b1d802505024d1db9f82beb783fa681: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:36:09.492203 27808 ts_tablet_manager.cc:1905] T ad82d8f5869b4eeb893181081f1c55e7 P 0f43868fca7f4100a11cb2ba2b1075ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:36:09.493046 27809 ts_tablet_manager.cc:1905] T ad82d8f5869b4eeb893181081f1c55e7 P 0b6e56e8210041b1ad2ec22acf97474c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:36:09.498122 27807 ts_tablet_manager.cc:1918] T ad82d8f5869b4eeb893181081f1c55e7 P 4b1d802505024d1db9f82beb783fa681: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:36:09.498301 27807 log.cc:1196] T ad82d8f5869b4eeb893181081f1c55e7 P 4b1d802505024d1db9f82beb783fa681: Deleting WAL directory at /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980527431848-22676-0/minicluster-data/ts-1/wal/wals/ad82d8f5869b4eeb893181081f1c55e7
I20260620 18:36:09.498804 27807 ts_tablet_manager.cc:1939] T ad82d8f5869b4eeb893181081f1c55e7 P 4b1d802505024d1db9f82beb783fa681: Deleting consensus metadata
I20260620 18:36:09.499260 27808 ts_tablet_manager.cc:1918] T ad82d8f5869b4eeb893181081f1c55e7 P 0f43868fca7f4100a11cb2ba2b1075ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:36:09.499420 27808 log.cc:1196] T ad82d8f5869b4eeb893181081f1c55e7 P 0f43868fca7f4100a11cb2ba2b1075ae: Deleting WAL directory at /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980527431848-22676-0/minicluster-data/ts-0/wal/wals/ad82d8f5869b4eeb893181081f1c55e7
I20260620 18:36:09.499949 27808 ts_tablet_manager.cc:1939] T ad82d8f5869b4eeb893181081f1c55e7 P 0f43868fca7f4100a11cb2ba2b1075ae: Deleting consensus metadata
I20260620 18:36:09.500713 27305 catalog_manager.cc:4841] TS 4b1d802505024d1db9f82beb783fa681 (127.22.37.2:35113): tablet ad82d8f5869b4eeb893181081f1c55e7 (table test-table [id=5bd9822ef57e4b55820abd6fdf9e570f]) successfully deleted
I20260620 18:36:09.501752 27809 ts_tablet_manager.cc:1918] T ad82d8f5869b4eeb893181081f1c55e7 P 0b6e56e8210041b1ad2ec22acf97474c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:36:09.501964 27809 log.cc:1196] T ad82d8f5869b4eeb893181081f1c55e7 P 0b6e56e8210041b1ad2ec22acf97474c: Deleting WAL directory at /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980527431848-22676-0/minicluster-data/ts-2/wal/wals/ad82d8f5869b4eeb893181081f1c55e7
I20260620 18:36:09.502207 27306 catalog_manager.cc:4841] TS 0f43868fca7f4100a11cb2ba2b1075ae (127.22.37.1:41215): tablet ad82d8f5869b4eeb893181081f1c55e7 (table test-table [id=5bd9822ef57e4b55820abd6fdf9e570f]) successfully deleted
I20260620 18:36:09.502537 27809 ts_tablet_manager.cc:1939] T ad82d8f5869b4eeb893181081f1c55e7 P 0b6e56e8210041b1ad2ec22acf97474c: Deleting consensus metadata
I20260620 18:36:09.504462 27304 catalog_manager.cc:4841] TS 0b6e56e8210041b1ad2ec22acf97474c (127.22.37.3:44303): tablet ad82d8f5869b4eeb893181081f1c55e7 (table test-table [id=5bd9822ef57e4b55820abd6fdf9e570f]) successfully deleted
W20260620 18:36:09.504907 27304 catalog_manager.cc:4808] TS 0b6e56e8210041b1ad2ec22acf97474c (127.22.37.3:44303): delete failed for tablet ad82d8f5869b4eeb893181081f1c55e7 because the tablet was not found. No further retry: Not found: Tablet not found: ad82d8f5869b4eeb893181081f1c55e7
I20260620 18:36:09.505549 27779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:09.491460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42954 (local address 127.22.37.1:41215)
0620 18:36:09.491614 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:09.491622 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:09.503212 (+ 11590us) server_negotiation.cc:407] Connection header received
0620 18:36:09.503415 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:09.503419 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:09.503499 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:09.503621 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:09.504737 (+ 1116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:09.504743 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:09.504748 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:36:09.504767 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:09.504803 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:36:09.504818 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:09.504945 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:09.505154 (+ 209us) server_negotiation.cc:299] Negotiation successful
0620 18:36:09.505214 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 18:36:09.506486 27500 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:36:09.674544 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 27378
I20260620 18:36:09.691475 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 27509
I20260620 18:36:09.694511 27782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:09.692305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57882 (local address 127.22.37.2:35113)
0620 18:36:09.692492 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:09.692500 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:09.692521 (+ 21us) server_negotiation.cc:407] Connection header received
0620 18:36:09.692727 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:09.692732 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:09.692815 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:09.692922 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:09.693835 (+ 913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:09.693841 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:09.693846 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:36:09.693862 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:09.693899 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:36:09.693911 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:09.694049 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:09.694171 (+ 122us) server_negotiation.cc:299] Negotiation successful
0620 18:36:09.694230 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 18:36:09.695230 27631 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:36:09.783071 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 27509
I20260620 18:36:09.799825 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 27639
I20260620 18:36:09.803002 27783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:09.800662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41336 (local address 127.22.37.3:44303)
0620 18:36:09.800846 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:09.800854 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:09.800878 (+ 24us) server_negotiation.cc:407] Connection header received
0620 18:36:09.801140 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:09.801146 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:09.801220 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:09.801354 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:09.802355 (+ 1001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:09.802360 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:09.802363 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:36:09.802379 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:09.802407 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:36:09.802419 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:09.802579 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:09.802638 (+ 59us) server_negotiation.cc:299] Negotiation successful
0620 18:36:09.802697 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 18:36:09.803745 27761 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:36:09.922289 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 27639
I20260620 18:36:09.941043 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 27288
I20260620 18:36:09.942188 27349 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:36:10.047174 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 27288
2026-06-20T18:36:10Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2656 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260620 18:36:10.087837 22676 mini_oidc.cc:152] Starting JWKS server
I20260620 18:36:10.090420 22676 webserver.cc:460] Webserver started at https://127.0.0.1:39617/ using document root <none> and password file <none>
I20260620 18:36:10.090701 22676 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 18:36:10.091311 22676 webserver.cc:460] Webserver started at http://127.0.0.1:38559/ using document root <none> and password file <none>
2026-06-20T18:36:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:36:10Z Disabled control of system clock
I20260620 18:36:10.117815 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980527431848-22676-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980527431848-22676-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980527431848-22676-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.62:45831
--webserver_interface=127.22.37.62
--webserver_port=0
--builtin_ntp_servers=127.22.37.20:42035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.37.62:45831
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39617/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980527431848-22676-0/testchainca.cert with env {}
W20260620 18:36:10.432305 27820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:36:10.432616 27820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:36:10.432737 27820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:36:10.440086 27820 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:36:10.440208 27820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:36:10.440245 27820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:36:10.440311 27820 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:36:10.440356 27820 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 18:36:10.440416 27820 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 18:36:10.440456 27820 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39617/jwks.json
I20260620 18:36:10.450584 27820 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:42035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980527431848-22676-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980527431848-22676-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.37.62:45831
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.62:45831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39617/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.37.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980527431848-22676-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980527431848-22676-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:36:10.451869 27820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:36:10.453130 27820 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:36:10.465149 27825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:10.465189 27826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:10.465149 27828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:10.465955 27820 server_base.cc:1034] running on GCE node
I20260620 18:36:10.466715 27820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:36:10.468142 27820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:36:10.469405 27820 hybrid_clock.cc:648] HybridClock initialized: now 1781980570469328 us; error 49 us; skew 500 ppm
I20260620 18:36:10.469774 27820 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:36:10.471988 27820 webserver.cc:460] Webserver started at http://127.22.37.62:38587/ using document root <none> and password file <none>
I20260620 18:36:10.472669 27820 fs_manager.cc:359] Metadata directory not provided
I20260620 18:36:10.472806 27820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:36:10.473119 27820 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:36:10.476011 27820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980527431848-22676-0/minicluster-data/master-0/data/instance:
uuid: "8d919d4cc2ce4e5bb996782a22c6e3aa"
format_stamp: "Formatted at 2026-06-20 18:36:10 on dist-test-slave-5lgp"
server_key: "52d01a90507c1272c543fe8da97fe0c8"
server_key_iv: "d476df7e7a0376379748a3d63a5bd558"
server_key_version: "encryptionkey@0"
I20260620 18:36:10.476771 27820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980527431848-22676-0/minicluster-data/master-0/wal/instance:
uuid: "8d919d4cc2ce4e5bb996782a22c6e3aa"
format_stamp: "Formatted at 2026-06-20 18:36:10 on dist-test-slave-5lgp"
server_key: "52d01a90507c1272c543fe8da97fe0c8"
server_key_iv: "d476df7e7a0376379748a3d63a5bd558"
server_key_version: "encryptionkey@0"
I20260620 18:36:10.481781 27820 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 18:36:10.485047 27834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:10.486353 27820 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:36:10.486538 27820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980527431848-22676-0/minicluster-data/master-0/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980527431848-22676-0/minicluster-data/master-0/wal
uuid: "8d919d4cc2ce4e5bb996782a22c6e3aa"
format_stamp: "Formatted at 2026-06-20 18:36:10 on dist-test-slave-5lgp"
server_key: "52d01a90507c1272c543fe8da97fe0c8"
server_key_iv: "d476df7e7a0376379748a3d63a5bd558"
server_key_version: "encryptionkey@0"
I20260620 18:36:10.486747 27820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980527431848-22676-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980527431848-22676-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980527431848-22676-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:36:10.506910 27820 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:36:10.507689 27820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:36:10.508001 27820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:36:10.524813 27820 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.62:45831
I20260620 18:36:10.524834 27885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.62:45831 every 8 connection(s)
I20260620 18:36:10.526156 27820 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
I20260620 18:36:10.529799 27886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:36:10.534703 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 27820
I20260620 18:36:10.534948 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980527431848-22676-0/minicluster-data/master-0/wal/instance
I20260620 18:36:10.535565 22676 external_mini_cluster.cc:1428] Setting key 78fa30ba7a563858ef69d4a78355cae2
I20260620 18:36:10.540330 27886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d919d4cc2ce4e5bb996782a22c6e3aa: Bootstrap starting.
I20260620 18:36:10.543828 27886 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d919d4cc2ce4e5bb996782a22c6e3aa: Neither blocks nor log segments found. Creating new log.
I20260620 18:36:10.545069 27886 log.cc:824] T 00000000000000000000000000000000 P 8d919d4cc2ce4e5bb996782a22c6e3aa: Log is configured to *not* fsync() on all Append() calls
I20260620 18:36:10.548161 27886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d919d4cc2ce4e5bb996782a22c6e3aa: No bootstrap required, opened a new log
I20260620 18:36:10.549243 27889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:10.537499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42940 (local address 127.22.37.62:45831)
0620 18:36:10.538557 (+ 1058us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:10.538567 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:10.538611 (+ 44us) server_negotiation.cc:407] Connection header received
0620 18:36:10.540257 (+ 1646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:10.540280 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:10.540790 (+ 510us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:10.541119 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:10.541804 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:10.543065 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:10.544963 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:10.545851 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:10.546070 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:10.547123 (+ 1053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:10.547130 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:10.547143 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:36:10.547653 (+ 510us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:10.547723 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:36:10.547742 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:10.547826 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:10.548052 (+ 226us) server_negotiation.cc:299] Negotiation successful
0620 18:36:10.548371 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":409,"threads_started":1}
I20260620 18:36:10.553078 22676 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:36:10.557544 27886 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d919d4cc2ce4e5bb996782a22c6e3aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d919d4cc2ce4e5bb996782a22c6e3aa" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 45831 } }
I20260620 18:36:10.557847 27886 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d919d4cc2ce4e5bb996782a22c6e3aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:36:10.557912 27886 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d919d4cc2ce4e5bb996782a22c6e3aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d919d4cc2ce4e5bb996782a22c6e3aa, State: Initialized, Role: FOLLOWER
I20260620 18:36:10.558350 27886 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d919d4cc2ce4e5bb996782a22c6e3aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d919d4cc2ce4e5bb996782a22c6e3aa" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 45831 } }
I20260620 18:36:10.558533 27886 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d919d4cc2ce4e5bb996782a22c6e3aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:36:10.558624 27886 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d919d4cc2ce4e5bb996782a22c6e3aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:36:10.558732 27886 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d919d4cc2ce4e5bb996782a22c6e3aa [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:36:10.560730 27886 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d919d4cc2ce4e5bb996782a22c6e3aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d919d4cc2ce4e5bb996782a22c6e3aa" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 45831 } }
I20260620 18:36:10.561120 27886 leader_election.cc:302] T 00000000000000000000000000000000 P 8d919d4cc2ce4e5bb996782a22c6e3aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d919d4cc2ce4e5bb996782a22c6e3aa; no voters:
I20260620 18:36:10.561728 27886 leader_election.cc:288] T 00000000000000000000000000000000 P 8d919d4cc2ce4e5bb996782a22c6e3aa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:36:10.561882 27891 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d919d4cc2ce4e5bb996782a22c6e3aa [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:36:10.562904 27891 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d919d4cc2ce4e5bb996782a22c6e3aa [term 1 LEADER]: Becoming Leader. State: Replica: 8d919d4cc2ce4e5bb996782a22c6e3aa, State: Running, Role: LEADER
I20260620 18:36:10.563405 27891 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d919d4cc2ce4e5bb996782a22c6e3aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d919d4cc2ce4e5bb996782a22c6e3aa" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 45831 } }
I20260620 18:36:10.563802 27886 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d919d4cc2ce4e5bb996782a22c6e3aa [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:36:10.566331 27892 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d919d4cc2ce4e5bb996782a22c6e3aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d919d4cc2ce4e5bb996782a22c6e3aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d919d4cc2ce4e5bb996782a22c6e3aa" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 45831 } } }
I20260620 18:36:10.566303 27893 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d919d4cc2ce4e5bb996782a22c6e3aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d919d4cc2ce4e5bb996782a22c6e3aa. Latest consensus state: current_term: 1 leader_uuid: "8d919d4cc2ce4e5bb996782a22c6e3aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d919d4cc2ce4e5bb996782a22c6e3aa" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 45831 } } }
I20260620 18:36:10.566859 27892 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d919d4cc2ce4e5bb996782a22c6e3aa [sys.catalog]: This master's current role is: LEADER
I20260620 18:36:10.566982 27893 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d919d4cc2ce4e5bb996782a22c6e3aa [sys.catalog]: This master's current role is: LEADER
I20260620 18:36:10.568573 27896 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:36:10.573911 27896 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:36:10.581250 27896 catalog_manager.cc:1347] Generated new cluster ID: a14b53c9df7348bbbc474ee627a781f1
I20260620 18:36:10.581357 27896 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:36:10.600169 27896 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:36:10.601477 27896 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:36:10.612042 27896 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d919d4cc2ce4e5bb996782a22c6e3aa: Generated new TSK 0
I20260620 18:36:10.612892 27896 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 18:36:10.655378 27916 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260620 18:36:10.655555 27916 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 18:36:10.656757 27889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:10.634279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42946 (local address 127.22.37.62:45831)
0620 18:36:10.634479 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:10.634485 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:10.635069 (+ 584us) server_negotiation.cc:407] Connection header received
0620 18:36:10.635345 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:10.635351 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:10.635438 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:10.635596 (+ 158us) server_negotiation.cc:226] Negotiated authn=JWT
0620 18:36:10.636089 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:10.637280 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:10.638794 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:10.639460 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:10.639583 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:10.640078 (+ 495us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 18:36:10.656054 (+ 15976us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39617/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 18:36:10.656457 (+ 403us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42946: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39617/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":70}
W20260620 18:36:10.656891 27889 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42946: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39617/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260620 18:36:10.658233 27915 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:36:10.634275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:45831 (local address 127.0.0.1:42946)
0620 18:36:10.634964 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:10.634983 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:10.635128 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:10.635618 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:10.635622 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:10.635632 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0620 18:36:10.635946 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:10.635954 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:10.637480 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:10.637484 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:10.638654 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:10.638665 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:10.639627 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:10.639632 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:10.639754 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:10.639810 (+ 56us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 18:36:10.656387 (+ 16577us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39617/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 18:36:10.656684 (+ 297us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.22.37.62:45831: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39617/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":360,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.22.37.62:45831: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39617/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260620 18:36:10.660116 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 27820
I20260620 18:36:10.661286 27881 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:36:10.756021 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 27820
2026-06-20T18:36:10Z chronyd exiting
I20260620 18:36:10.806036 22676 test_util.cc:175] -----------------------------------------------
I20260620 18:36:10.806206 22676 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980527431848-22676-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (722 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260620 18:36:10.809826 22676 mini_oidc.cc:152] Starting JWKS server
I20260620 18:36:10.811653 22676 webserver.cc:460] Webserver started at https://127.0.0.1:33835/ using document root <none> and password file <none>
I20260620 18:36:10.811889 22676 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 18:36:10.812489 22676 webserver.cc:460] Webserver started at http://127.0.0.1:34235/ using document root <none> and password file <none>
2026-06-20T18:36:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:36:10Z Disabled control of system clock
I20260620 18:36:10.839039 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980527431848-22676-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980527431848-22676-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980527431848-22676-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.62:45939
--webserver_interface=127.22.37.62
--webserver_port=0
--builtin_ntp_servers=127.22.37.20:37283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.37.62:45939
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33835/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980527431848-22676-0/testchainca.cert with env {}
W20260620 18:36:11.155503 27923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:36:11.155772 27923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:36:11.155855 27923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:36:11.162881 27923 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:36:11.162978 27923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:36:11.163015 27923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:36:11.163064 27923 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:36:11.163105 27923 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 18:36:11.163165 27923 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 18:36:11.163198 27923 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33835/jwks.json
I20260620 18:36:11.172369 27923 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:37283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980527431848-22676-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980527431848-22676-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.37.62:45939
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.62:45939
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33835/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.37.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980527431848-22676-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980527431848-22676-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:36:11.173542 27923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:36:11.174733 27923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:36:11.185678 27928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:11.185796 27929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:11.186187 27931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:11.186869 27923 server_base.cc:1034] running on GCE node
I20260620 18:36:11.187598 27923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:36:11.188822 27923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:36:11.190008 27923 hybrid_clock.cc:648] HybridClock initialized: now 1781980571189971 us; error 37 us; skew 500 ppm
I20260620 18:36:11.190340 27923 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:36:11.192476 27923 webserver.cc:460] Webserver started at http://127.22.37.62:38235/ using document root <none> and password file <none>
I20260620 18:36:11.193097 27923 fs_manager.cc:359] Metadata directory not provided
I20260620 18:36:11.193208 27923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:36:11.193570 27923 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:36:11.196276 27923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980527431848-22676-0/minicluster-data/master-0/data/instance:
uuid: "b47c813b73a940fbabf9669cb7451871"
format_stamp: "Formatted at 2026-06-20 18:36:11 on dist-test-slave-5lgp"
server_key: "23e7ce334c8260740f7415063dea2253"
server_key_iv: "04a2a114f3799be414d34d3a1e6a07db"
server_key_version: "encryptionkey@0"
I20260620 18:36:11.197013 27923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980527431848-22676-0/minicluster-data/master-0/wal/instance:
uuid: "b47c813b73a940fbabf9669cb7451871"
format_stamp: "Formatted at 2026-06-20 18:36:11 on dist-test-slave-5lgp"
server_key: "23e7ce334c8260740f7415063dea2253"
server_key_iv: "04a2a114f3799be414d34d3a1e6a07db"
server_key_version: "encryptionkey@0"
I20260620 18:36:11.201642 27923 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260620 18:36:11.204905 27937 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:11.206157 27923 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:36:11.206336 27923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980527431848-22676-0/minicluster-data/master-0/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980527431848-22676-0/minicluster-data/master-0/wal
uuid: "b47c813b73a940fbabf9669cb7451871"
format_stamp: "Formatted at 2026-06-20 18:36:11 on dist-test-slave-5lgp"
server_key: "23e7ce334c8260740f7415063dea2253"
server_key_iv: "04a2a114f3799be414d34d3a1e6a07db"
server_key_version: "encryptionkey@0"
I20260620 18:36:11.206532 27923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980527431848-22676-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980527431848-22676-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980527431848-22676-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:36:11.233067 27923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:36:11.233933 27923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:36:11.234261 27923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:36:11.252174 27923 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.62:45939
I20260620 18:36:11.252193 27988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.62:45939 every 8 connection(s)
I20260620 18:36:11.253504 27923 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
I20260620 18:36:11.256868 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 27923
I20260620 18:36:11.257066 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980527431848-22676-0/minicluster-data/master-0/wal/instance
I20260620 18:36:11.257129 27989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:36:11.257788 22676 external_mini_cluster.cc:1428] Setting key 09cde41966a84a5e255e3f2c17c00879
I20260620 18:36:11.270399 27989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b47c813b73a940fbabf9669cb7451871: Bootstrap starting.
I20260620 18:36:11.271956 27992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:11.259732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47680 (local address 127.22.37.62:45939)
0620 18:36:11.260727 (+ 995us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:11.260744 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:11.260789 (+ 45us) server_negotiation.cc:407] Connection header received
0620 18:36:11.262922 (+ 2133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:11.262943 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:11.263390 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:11.263718 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:11.264291 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:11.265601 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:11.267417 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:11.268104 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:11.268235 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:11.269558 (+ 1323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:11.269566 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:11.269579 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:36:11.270107 (+ 528us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:11.270180 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:36:11.270199 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:11.270305 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:11.270591 (+ 286us) server_negotiation.cc:299] Negotiation successful
0620 18:36:11.270928 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":384,"threads_started":1}
I20260620 18:36:11.273757 27989 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b47c813b73a940fbabf9669cb7451871: Neither blocks nor log segments found. Creating new log.
I20260620 18:36:11.274951 27989 log.cc:824] T 00000000000000000000000000000000 P b47c813b73a940fbabf9669cb7451871: Log is configured to *not* fsync() on all Append() calls
I20260620 18:36:11.275667 22676 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:36:11.277710 27989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b47c813b73a940fbabf9669cb7451871: No bootstrap required, opened a new log
I20260620 18:36:11.284596 27989 raft_consensus.cc:348] T 00000000000000000000000000000000 P b47c813b73a940fbabf9669cb7451871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b47c813b73a940fbabf9669cb7451871" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 45939 } }
I20260620 18:36:11.284948 27989 raft_consensus.cc:374] T 00000000000000000000000000000000 P b47c813b73a940fbabf9669cb7451871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:36:11.285089 27989 raft_consensus.cc:729] T 00000000000000000000000000000000 P b47c813b73a940fbabf9669cb7451871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b47c813b73a940fbabf9669cb7451871, State: Initialized, Role: FOLLOWER
I20260620 18:36:11.285704 27989 consensus_queue.cc:260] T 00000000000000000000000000000000 P b47c813b73a940fbabf9669cb7451871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b47c813b73a940fbabf9669cb7451871" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 45939 } }
I20260620 18:36:11.285943 27989 raft_consensus.cc:388] T 00000000000000000000000000000000 P b47c813b73a940fbabf9669cb7451871 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:36:11.286088 27989 raft_consensus.cc:482] T 00000000000000000000000000000000 P b47c813b73a940fbabf9669cb7451871 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:36:11.286252 27989 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b47c813b73a940fbabf9669cb7451871 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:36:11.288348 27989 raft_consensus.cc:504] T 00000000000000000000000000000000 P b47c813b73a940fbabf9669cb7451871 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b47c813b73a940fbabf9669cb7451871" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 45939 } }
I20260620 18:36:11.288892 27989 leader_election.cc:302] T 00000000000000000000000000000000 P b47c813b73a940fbabf9669cb7451871 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b47c813b73a940fbabf9669cb7451871; no voters:
I20260620 18:36:11.289547 27989 leader_election.cc:288] T 00000000000000000000000000000000 P b47c813b73a940fbabf9669cb7451871 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:36:11.289636 27994 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b47c813b73a940fbabf9669cb7451871 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:36:11.290409 27994 raft_consensus.cc:686] T 00000000000000000000000000000000 P b47c813b73a940fbabf9669cb7451871 [term 1 LEADER]: Becoming Leader. State: Replica: b47c813b73a940fbabf9669cb7451871, State: Running, Role: LEADER
I20260620 18:36:11.290901 27994 consensus_queue.cc:237] T 00000000000000000000000000000000 P b47c813b73a940fbabf9669cb7451871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b47c813b73a940fbabf9669cb7451871" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 45939 } }
I20260620 18:36:11.291640 27989 sys_catalog.cc:564] T 00000000000000000000000000000000 P b47c813b73a940fbabf9669cb7451871 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:36:11.293892 27995 sys_catalog.cc:455] T 00000000000000000000000000000000 P b47c813b73a940fbabf9669cb7451871 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b47c813b73a940fbabf9669cb7451871" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b47c813b73a940fbabf9669cb7451871" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 45939 } } }
I20260620 18:36:11.294299 27995 sys_catalog.cc:458] T 00000000000000000000000000000000 P b47c813b73a940fbabf9669cb7451871 [sys.catalog]: This master's current role is: LEADER
I20260620 18:36:11.294970 27996 sys_catalog.cc:455] T 00000000000000000000000000000000 P b47c813b73a940fbabf9669cb7451871 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b47c813b73a940fbabf9669cb7451871. Latest consensus state: current_term: 1 leader_uuid: "b47c813b73a940fbabf9669cb7451871" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b47c813b73a940fbabf9669cb7451871" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 45939 } } }
I20260620 18:36:11.295511 27996 sys_catalog.cc:458] T 00000000000000000000000000000000 P b47c813b73a940fbabf9669cb7451871 [sys.catalog]: This master's current role is: LEADER
I20260620 18:36:11.302016 27999 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:36:11.307586 27999 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:36:11.316679 27999 catalog_manager.cc:1347] Generated new cluster ID: 1db0f14e61e748ce8f41ed4f90c10541
I20260620 18:36:11.316808 27999 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:36:11.345629 27999 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:36:11.346942 27999 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:36:11.359124 27999 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b47c813b73a940fbabf9669cb7451871: Generated new TSK 0
I20260620 18:36:11.359995 27999 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 18:36:11.406395 28019 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260620 18:36:11.406530 28019 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 18:36:11.407462 27992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:11.379695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47692 (local address 127.22.37.62:45939)
0620 18:36:11.380252 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:11.380259 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:11.380340 (+ 81us) server_negotiation.cc:407] Connection header received
0620 18:36:11.380584 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:11.380589 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:11.380676 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:11.380803 (+ 127us) server_negotiation.cc:226] Negotiated authn=JWT
0620 18:36:11.381421 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:11.382162 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:11.384205 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:11.384678 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:11.384766 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:11.385460 (+ 694us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 18:36:11.406919 (+ 21459us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33835/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 18:36:11.407239 (+ 320us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47692: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33835/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":438}
W20260620 18:36:11.407616 27992 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47692: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33835/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260620 18:36:11.407871 28018 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:36:11.379522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:45939 (local address 127.0.0.1:47692)
0620 18:36:11.380232 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:11.380252 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:11.380407 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:11.380849 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:11.380854 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:11.380867 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0620 18:36:11.381213 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:11.381226 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:11.382320 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:11.382327 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:11.384055 (+ 1728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:11.384073 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:11.384852 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:11.384859 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:11.385103 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:11.385154 (+ 51us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 18:36:11.407190 (+ 22036us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33835/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 18:36:11.407478 (+ 288us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.22.37.62:45939: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33835/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":377,"threads_started":1}
I20260620 18:36:11.408903 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 27923
I20260620 18:36:11.410059 27984 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:36:11.502398 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 27923
2026-06-20T18:36:11Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (747 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260620 18:36:11.557456 22676 mini_oidc.cc:152] Starting JWKS server
I20260620 18:36:11.559216 22676 webserver.cc:460] Webserver started at https://127.0.0.1:45101/ using document root <none> and password file <none>
I20260620 18:36:11.559417 22676 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 18:36:11.559907 22676 webserver.cc:460] Webserver started at http://127.0.0.1:39121/ using document root <none> and password file <none>
2026-06-20T18:36:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:36:11Z Disabled control of system clock
I20260620 18:36:11.585878 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980527431848-22676-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980527431848-22676-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980527431848-22676-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.62:43379
--webserver_interface=127.22.37.62
--webserver_port=0
--builtin_ntp_servers=127.22.37.20:36765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.37.62:43379
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45101/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 18:36:11.898718 28026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:36:11.899019 28026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:36:11.899104 28026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:36:11.906405 28026 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:36:11.906522 28026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:36:11.906559 28026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:36:11.906607 28026 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:36:11.906651 28026 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 18:36:11.906703 28026 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 18:36:11.906740 28026 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45101/jwks.json
I20260620 18:36:11.915711 28026 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:36765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980527431848-22676-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980527431848-22676-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.37.62:43379
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.62:43379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45101/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.37.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980527431848-22676-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:36:11.916981 28026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:36:11.918718 28026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:36:11.930572 28031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:11.930620 28032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:11.930639 28034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:11.931545 28026 server_base.cc:1034] running on GCE node
I20260620 18:36:11.932407 28026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:36:11.933959 28026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:36:11.935163 28026 hybrid_clock.cc:648] HybridClock initialized: now 1781980571935105 us; error 58 us; skew 500 ppm
I20260620 18:36:11.935534 28026 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:36:11.937848 28026 webserver.cc:460] Webserver started at http://127.22.37.62:35997/ using document root <none> and password file <none>
I20260620 18:36:11.938558 28026 fs_manager.cc:359] Metadata directory not provided
I20260620 18:36:11.938671 28026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:36:11.939014 28026 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:36:11.942025 28026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980527431848-22676-0/minicluster-data/master-0/data/instance:
uuid: "093bac97b8b643d3bcb4ebc9d9e60999"
format_stamp: "Formatted at 2026-06-20 18:36:11 on dist-test-slave-5lgp"
server_key: "5ec14858ab857ab4298885adb00e3316"
server_key_iv: "095c9cc5a606cd072ec43db3fae4643b"
server_key_version: "encryptionkey@0"
I20260620 18:36:11.942837 28026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980527431848-22676-0/minicluster-data/master-0/wal/instance:
uuid: "093bac97b8b643d3bcb4ebc9d9e60999"
format_stamp: "Formatted at 2026-06-20 18:36:11 on dist-test-slave-5lgp"
server_key: "5ec14858ab857ab4298885adb00e3316"
server_key_iv: "095c9cc5a606cd072ec43db3fae4643b"
server_key_version: "encryptionkey@0"
I20260620 18:36:11.947858 28026 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:36:11.951287 28040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:11.952594 28026 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:36:11.952782 28026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980527431848-22676-0/minicluster-data/master-0/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980527431848-22676-0/minicluster-data/master-0/wal
uuid: "093bac97b8b643d3bcb4ebc9d9e60999"
format_stamp: "Formatted at 2026-06-20 18:36:11 on dist-test-slave-5lgp"
server_key: "5ec14858ab857ab4298885adb00e3316"
server_key_iv: "095c9cc5a606cd072ec43db3fae4643b"
server_key_version: "encryptionkey@0"
I20260620 18:36:11.952999 28026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980527431848-22676-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980527431848-22676-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980527431848-22676-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:36:11.976078 28026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:36:11.976940 28026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:36:11.977249 28026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:36:11.995206 28026 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.62:43379
I20260620 18:36:11.995213 28091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.62:43379 every 8 connection(s)
I20260620 18:36:11.996630 28026 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
I20260620 18:36:12.000422 28092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:36:12.002416 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 28026
I20260620 18:36:12.002636 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980527431848-22676-0/minicluster-data/master-0/wal/instance
I20260620 18:36:12.003244 22676 external_mini_cluster.cc:1428] Setting key 74eb627281af509e03a2af879a24193c
I20260620 18:36:12.013015 28092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 093bac97b8b643d3bcb4ebc9d9e60999: Bootstrap starting.
I20260620 18:36:12.016381 28092 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 093bac97b8b643d3bcb4ebc9d9e60999: Neither blocks nor log segments found. Creating new log.
I20260620 18:36:12.016798 28095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:12.004932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57014 (local address 127.22.37.62:43379)
0620 18:36:12.006074 (+ 1142us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:12.006087 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:12.006132 (+ 45us) server_negotiation.cc:407] Connection header received
0620 18:36:12.007798 (+ 1666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:12.007821 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:12.008279 (+ 458us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:12.008597 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:12.009206 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:12.010634 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:12.012407 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:12.013058 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:12.013255 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:12.014495 (+ 1240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:12.014503 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:12.014517 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:36:12.015066 (+ 549us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:12.015153 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:36:12.015173 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:12.015304 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:12.015518 (+ 214us) server_negotiation.cc:299] Negotiation successful
0620 18:36:12.015826 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":625,"thread_start_us":383,"threads_started":1}
I20260620 18:36:12.017697 28092 log.cc:824] T 00000000000000000000000000000000 P 093bac97b8b643d3bcb4ebc9d9e60999: Log is configured to *not* fsync() on all Append() calls
I20260620 18:36:12.020931 22676 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:36:12.021135 28092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 093bac97b8b643d3bcb4ebc9d9e60999: No bootstrap required, opened a new log
I20260620 18:36:12.029479 28092 raft_consensus.cc:348] T 00000000000000000000000000000000 P 093bac97b8b643d3bcb4ebc9d9e60999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "093bac97b8b643d3bcb4ebc9d9e60999" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 43379 } }
I20260620 18:36:12.029908 28092 raft_consensus.cc:374] T 00000000000000000000000000000000 P 093bac97b8b643d3bcb4ebc9d9e60999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:36:12.030043 28092 raft_consensus.cc:729] T 00000000000000000000000000000000 P 093bac97b8b643d3bcb4ebc9d9e60999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 093bac97b8b643d3bcb4ebc9d9e60999, State: Initialized, Role: FOLLOWER
I20260620 18:36:12.030733 28092 consensus_queue.cc:260] T 00000000000000000000000000000000 P 093bac97b8b643d3bcb4ebc9d9e60999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "093bac97b8b643d3bcb4ebc9d9e60999" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 43379 } }
I20260620 18:36:12.030992 28092 raft_consensus.cc:388] T 00000000000000000000000000000000 P 093bac97b8b643d3bcb4ebc9d9e60999 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:36:12.031149 28092 raft_consensus.cc:482] T 00000000000000000000000000000000 P 093bac97b8b643d3bcb4ebc9d9e60999 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:36:12.031338 28092 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 093bac97b8b643d3bcb4ebc9d9e60999 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:36:12.033968 28092 raft_consensus.cc:504] T 00000000000000000000000000000000 P 093bac97b8b643d3bcb4ebc9d9e60999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "093bac97b8b643d3bcb4ebc9d9e60999" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 43379 } }
I20260620 18:36:12.034535 28092 leader_election.cc:302] T 00000000000000000000000000000000 P 093bac97b8b643d3bcb4ebc9d9e60999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 093bac97b8b643d3bcb4ebc9d9e60999; no voters:
I20260620 18:36:12.035156 28092 leader_election.cc:288] T 00000000000000000000000000000000 P 093bac97b8b643d3bcb4ebc9d9e60999 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:36:12.035300 28097 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 093bac97b8b643d3bcb4ebc9d9e60999 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:36:12.036068 28097 raft_consensus.cc:686] T 00000000000000000000000000000000 P 093bac97b8b643d3bcb4ebc9d9e60999 [term 1 LEADER]: Becoming Leader. State: Replica: 093bac97b8b643d3bcb4ebc9d9e60999, State: Running, Role: LEADER
I20260620 18:36:12.036623 28097 consensus_queue.cc:237] T 00000000000000000000000000000000 P 093bac97b8b643d3bcb4ebc9d9e60999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "093bac97b8b643d3bcb4ebc9d9e60999" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 43379 } }
I20260620 18:36:12.037515 28092 sys_catalog.cc:564] T 00000000000000000000000000000000 P 093bac97b8b643d3bcb4ebc9d9e60999 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:36:12.039462 28098 sys_catalog.cc:455] T 00000000000000000000000000000000 P 093bac97b8b643d3bcb4ebc9d9e60999 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "093bac97b8b643d3bcb4ebc9d9e60999" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "093bac97b8b643d3bcb4ebc9d9e60999" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 43379 } } }
I20260620 18:36:12.039979 28098 sys_catalog.cc:458] T 00000000000000000000000000000000 P 093bac97b8b643d3bcb4ebc9d9e60999 [sys.catalog]: This master's current role is: LEADER
I20260620 18:36:12.039989 28099 sys_catalog.cc:455] T 00000000000000000000000000000000 P 093bac97b8b643d3bcb4ebc9d9e60999 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 093bac97b8b643d3bcb4ebc9d9e60999. Latest consensus state: current_term: 1 leader_uuid: "093bac97b8b643d3bcb4ebc9d9e60999" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "093bac97b8b643d3bcb4ebc9d9e60999" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 43379 } } }
I20260620 18:36:12.040874 28099 sys_catalog.cc:458] T 00000000000000000000000000000000 P 093bac97b8b643d3bcb4ebc9d9e60999 [sys.catalog]: This master's current role is: LEADER
I20260620 18:36:12.044072 28103 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:36:12.048446 28103 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:36:12.057083 28103 catalog_manager.cc:1347] Generated new cluster ID: 4c5408f5ff4b4231b72b8e557626105a
I20260620 18:36:12.057204 28103 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:36:12.067452 28103 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:36:12.069314 28103 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:36:12.082221 28103 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 093bac97b8b643d3bcb4ebc9d9e60999: Generated new TSK 0
I20260620 18:36:12.083056 28103 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:36:12.111382 28095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:12.103022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57018 (local address 127.22.37.62:43379)
0620 18:36:12.103241 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:12.103249 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:12.103612 (+ 363us) server_negotiation.cc:407] Connection header received
0620 18:36:12.103847 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:12.103851 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:12.103916 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:12.104033 (+ 117us) server_negotiation.cc:226] Negotiated authn=JWT
0620 18:36:12.104588 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:12.105284 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:12.107293 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:12.107719 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:12.107823 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:12.108414 (+ 591us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 18:36:12.110765 (+ 2351us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45101/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 18:36:12.111100 (+ 335us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57018: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45101/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":90}
W20260620 18:36:12.111510 28095 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57018: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45101/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260620 18:36:12.113492 28121 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:36:12.102853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:43379 (local address 127.0.0.1:57018)
0620 18:36:12.103504 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:12.103523 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:12.103676 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:12.104079 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:12.104086 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:12.104104 (+ 18us) client_negotiation.cc:189] Negotiated authn=JWT
0620 18:36:12.104437 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:12.104453 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:12.105450 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:12.105456 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:12.107158 (+ 1702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:12.107176 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:12.107919 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:12.107926 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:12.108116 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:12.108164 (+ 48us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 18:36:12.110979 (+ 2815us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45101/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 18:36:12.113034 (+ 2055us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.22.37.62:43379: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45101/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":498,"mutex_wait_us":1736,"thread_start_us":316,"threads_started":1}
I20260620 18:36:12.114823 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 28026
I20260620 18:36:12.116006 28087 generic_service.cc:196] Checking for leaks (request via RPC)
E20260620 18:36:12.122233 28122 webserver.cc:533] Webserver: sslize failed:
E20260620 18:36:12.122345 28122 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 18:36:12.213158 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 28026
2026-06-20T18:36:12Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (710 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-20T18:36:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:36:12Z Disabled control of system clock
I20260620 18:36:12.294308 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980527431848-22676-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980527431848-22676-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980527431848-22676-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.62:33627
--webserver_interface=127.22.37.62
--webserver_port=0
--builtin_ntp_servers=127.22.37.20:42981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.37.62:33627
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980527431848-22676-0/testchainca.cert with env {}
W20260620 18:36:12.609318 28127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:36:12.609609 28127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:36:12.609695 28127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:36:12.616616 28127 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:36:12.616720 28127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:36:12.616756 28127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:36:12.616804 28127 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:36:12.616847 28127 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 18:36:12.616904 28127 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 18:36:12.616945 28127 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260620 18:36:12.626487 28127 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:42981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980527431848-22676-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980527431848-22676-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.37.62:33627
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.62:33627
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.37.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980527431848-22676-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980527431848-22676-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:36:12.627660 28127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:36:12.628909 28127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:36:12.640774 28133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:12.641104 28132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:12.641116 28135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:12.642259 28127 server_base.cc:1034] running on GCE node
I20260620 18:36:12.643029 28127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:36:12.644439 28127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:36:12.645658 28127 hybrid_clock.cc:648] HybridClock initialized: now 1781980572645599 us; error 57 us; skew 500 ppm
I20260620 18:36:12.646004 28127 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:36:12.648244 28127 webserver.cc:460] Webserver started at http://127.22.37.62:39579/ using document root <none> and password file <none>
I20260620 18:36:12.648952 28127 fs_manager.cc:359] Metadata directory not provided
I20260620 18:36:12.649091 28127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:36:12.649451 28127 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:36:12.652341 28127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980527431848-22676-0/minicluster-data/master-0/data/instance:
uuid: "ec5dc33878fd48e5a611aed8cf0426a1"
format_stamp: "Formatted at 2026-06-20 18:36:12 on dist-test-slave-5lgp"
server_key: "a034b98f5ace84844174d37bdd950a1b"
server_key_iv: "140e5aff0dc3bc086b74bf0cad103c98"
server_key_version: "encryptionkey@0"
I20260620 18:36:12.653147 28127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980527431848-22676-0/minicluster-data/master-0/wal/instance:
uuid: "ec5dc33878fd48e5a611aed8cf0426a1"
format_stamp: "Formatted at 2026-06-20 18:36:12 on dist-test-slave-5lgp"
server_key: "a034b98f5ace84844174d37bdd950a1b"
server_key_iv: "140e5aff0dc3bc086b74bf0cad103c98"
server_key_version: "encryptionkey@0"
I20260620 18:36:12.658299 28127 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260620 18:36:12.661581 28141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:12.662863 28127 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 18:36:12.663038 28127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980527431848-22676-0/minicluster-data/master-0/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980527431848-22676-0/minicluster-data/master-0/wal
uuid: "ec5dc33878fd48e5a611aed8cf0426a1"
format_stamp: "Formatted at 2026-06-20 18:36:12 on dist-test-slave-5lgp"
server_key: "a034b98f5ace84844174d37bdd950a1b"
server_key_iv: "140e5aff0dc3bc086b74bf0cad103c98"
server_key_version: "encryptionkey@0"
I20260620 18:36:12.663257 28127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980527431848-22676-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980527431848-22676-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980527431848-22676-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:36:12.685184 28127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:36:12.686092 28127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:36:12.686401 28127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:36:12.704702 28127 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.62:33627
I20260620 18:36:12.704711 28192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.62:33627 every 8 connection(s)
I20260620 18:36:12.706048 28127 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
I20260620 18:36:12.709959 28193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:36:12.712246 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 28127
I20260620 18:36:12.712430 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980527431848-22676-0/minicluster-data/master-0/wal/instance
I20260620 18:36:12.713092 22676 external_mini_cluster.cc:1428] Setting key 8a1e93a570e4aeae6b5ef951f7bf2031
I20260620 18:36:12.720913 28193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec5dc33878fd48e5a611aed8cf0426a1: Bootstrap starting.
I20260620 18:36:12.723913 28193 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec5dc33878fd48e5a611aed8cf0426a1: Neither blocks nor log segments found. Creating new log.
I20260620 18:36:12.725006 28193 log.cc:824] T 00000000000000000000000000000000 P ec5dc33878fd48e5a611aed8cf0426a1: Log is configured to *not* fsync() on all Append() calls
I20260620 18:36:12.726243 28196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:12.715064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56940 (local address 127.22.37.62:33627)
0620 18:36:12.715980 (+ 916us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:12.715991 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:12.716028 (+ 37us) server_negotiation.cc:407] Connection header received
0620 18:36:12.717661 (+ 1633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:12.717682 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:12.718142 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:12.718541 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:12.719099 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:12.720307 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:12.721672 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:12.722338 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:12.722537 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:12.723685 (+ 1148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:12.723692 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:12.723706 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:36:12.724321 (+ 615us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:12.724378 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:36:12.724397 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:12.724520 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:12.724749 (+ 229us) server_negotiation.cc:299] Negotiation successful
0620 18:36:12.725025 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":317,"threads_started":1}
I20260620 18:36:12.728093 28193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec5dc33878fd48e5a611aed8cf0426a1: No bootstrap required, opened a new log
I20260620 18:36:12.729967 22676 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:36:12.735817 28193 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec5dc33878fd48e5a611aed8cf0426a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec5dc33878fd48e5a611aed8cf0426a1" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 33627 } }
I20260620 18:36:12.736172 28193 raft_consensus.cc:374] T 00000000000000000000000000000000 P ec5dc33878fd48e5a611aed8cf0426a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:36:12.736276 28193 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec5dc33878fd48e5a611aed8cf0426a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec5dc33878fd48e5a611aed8cf0426a1, State: Initialized, Role: FOLLOWER
I20260620 18:36:12.736871 28193 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec5dc33878fd48e5a611aed8cf0426a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec5dc33878fd48e5a611aed8cf0426a1" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 33627 } }
I20260620 18:36:12.737107 28193 raft_consensus.cc:388] T 00000000000000000000000000000000 P ec5dc33878fd48e5a611aed8cf0426a1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:36:12.737243 28193 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec5dc33878fd48e5a611aed8cf0426a1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:36:12.737440 28193 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec5dc33878fd48e5a611aed8cf0426a1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:36:12.739467 28193 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec5dc33878fd48e5a611aed8cf0426a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec5dc33878fd48e5a611aed8cf0426a1" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 33627 } }
I20260620 18:36:12.739919 28193 leader_election.cc:302] T 00000000000000000000000000000000 P ec5dc33878fd48e5a611aed8cf0426a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec5dc33878fd48e5a611aed8cf0426a1; no voters:
I20260620 18:36:12.740475 28193 leader_election.cc:288] T 00000000000000000000000000000000 P ec5dc33878fd48e5a611aed8cf0426a1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:36:12.740625 28198 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec5dc33878fd48e5a611aed8cf0426a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:36:12.741710 28198 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec5dc33878fd48e5a611aed8cf0426a1 [term 1 LEADER]: Becoming Leader. State: Replica: ec5dc33878fd48e5a611aed8cf0426a1, State: Running, Role: LEADER
I20260620 18:36:12.742247 28198 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec5dc33878fd48e5a611aed8cf0426a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec5dc33878fd48e5a611aed8cf0426a1" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 33627 } }
I20260620 18:36:12.742666 28193 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec5dc33878fd48e5a611aed8cf0426a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:36:12.745015 28200 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec5dc33878fd48e5a611aed8cf0426a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec5dc33878fd48e5a611aed8cf0426a1. Latest consensus state: current_term: 1 leader_uuid: "ec5dc33878fd48e5a611aed8cf0426a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec5dc33878fd48e5a611aed8cf0426a1" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 33627 } } }
I20260620 18:36:12.745426 28200 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec5dc33878fd48e5a611aed8cf0426a1 [sys.catalog]: This master's current role is: LEADER
I20260620 18:36:12.745097 28199 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec5dc33878fd48e5a611aed8cf0426a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ec5dc33878fd48e5a611aed8cf0426a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec5dc33878fd48e5a611aed8cf0426a1" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 33627 } } }
I20260620 18:36:12.745848 28199 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec5dc33878fd48e5a611aed8cf0426a1 [sys.catalog]: This master's current role is: LEADER
I20260620 18:36:12.746502 28207 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:36:12.752492 28207 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:36:12.763948 28207 catalog_manager.cc:1347] Generated new cluster ID: 360b57b1ce344dcdb30c297f4d855c4e
I20260620 18:36:12.764062 28207 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:36:12.777541 28207 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:36:12.778792 28207 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:36:12.787740 28207 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ec5dc33878fd48e5a611aed8cf0426a1: Generated new TSK 0
I20260620 18:36:12.788816 28207 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:36:12.835493 28196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:12.807585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56946 (local address 127.22.37.62:33627)
0620 18:36:12.807769 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:12.807775 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:12.808183 (+ 408us) server_negotiation.cc:407] Connection header received
0620 18:36:12.808364 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:12.808370 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:12.808465 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:12.808621 (+ 156us) server_negotiation.cc:226] Negotiated authn=JWT
0620 18:36:12.809020 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:12.810167 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:12.811481 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:12.812177 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:12.812292 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:12.812725 (+ 433us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 18:36:12.834738 (+ 22013us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 18:36:12.835157 (+ 419us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56946: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":67}
W20260620 18:36:12.835557 28222 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:36:12.807453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:33627 (local address 127.0.0.1:56946)
0620 18:36:12.808088 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:12.808103 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:12.808216 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:12.808629 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:12.808632 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:12.808641 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0620 18:36:12.808874 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:12.808881 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:12.810305 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:12.810308 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:12.811352 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:12.811363 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:12.812332 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:12.812336 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:12.812449 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:12.812478 (+ 29us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 18:36:12.835031 (+ 22553us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 18:36:12.835278 (+ 247us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.22.37.62:33627: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":367,"threads_started":1}
W20260620 18:36:12.835634 28196 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56946: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
I20260620 18:36:12.836297 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 28127
I20260620 18:36:12.837474 28188 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:36:12.930397 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 28127
2026-06-20T18:36:12Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (696 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (403 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (399 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980527431848-22676-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:36:13 dist-test-slave-5lgp krb5kdc[28237](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:36:13 dist-test-slave-5lgp krb5kdc[28237](info): set up 2 sockets
Jun 20 18:36:13 dist-test-slave-5lgp krb5kdc[28237](info): commencing operation
krb5kdc: starting...
W20260620 18:36:14.817951 22676 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.007s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980527431848-22676-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980527431848-22676-0/krb5kdc/test-user.keytab.
Jun 20 18:36:14 dist-test-slave-5lgp krb5kdc[28237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980574, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:36:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:36:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.37.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.37.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.37.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab.
Entry for principal kudu/127.22.37.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab.
I20260620 18:36:15.018919 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980527431848-22676-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980527431848-22676-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980527431848-22676-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.62:41883
--webserver_interface=127.22.37.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab
--principal=kudu/127.22.37.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:34181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.37.62:41883
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:36:15.335386 28250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:36:15.335664 28250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:36:15.335815 28250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:36:15.343066 28250 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:36:15.343175 28250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:36:15.343297 28250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:36:15.343376 28250 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:36:15.343413 28250 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:36:15.352924 28250 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:34181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980527431848-22676-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980527431848-22676-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.37.62:41883
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab
--principal=kudu/127.22.37.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.62:41883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.37.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980527431848-22676-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:36:15.354297 28250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:36:15.355618 28250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:36:15.367619 28255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:15.367666 28258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:15.367715 28256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:15.368623 28250 server_base.cc:1034] running on GCE node
I20260620 18:36:15.369439 28250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:36:15.370882 28250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:36:15.372136 28250 hybrid_clock.cc:648] HybridClock initialized: now 1781980575372080 us; error 52 us; skew 500 ppm
I20260620 18:36:15.372485 28250 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:36:15 dist-test-slave-5lgp krb5kdc[28237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980575, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:36:15.375526 28250 init.cc:377] Logged in from keytab as kudu/127.22.37.62@KRBTEST.COM (short username kudu)
I20260620 18:36:15.377018 28250 webserver.cc:460] Webserver started at http://127.22.37.62:40769/ using document root <none> and password file <none>
I20260620 18:36:15.377805 28250 fs_manager.cc:359] Metadata directory not provided
I20260620 18:36:15.377933 28250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:36:15.378269 28250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:36:15.381934 28250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980527431848-22676-0/minicluster-data/master-0/data/instance:
uuid: "4fc01f70031848de981f25099f8a0355"
format_stamp: "Formatted at 2026-06-20 18:36:15 on dist-test-slave-5lgp"
server_key: "f1f607e75ffc889ccb9bf652734e53d1"
server_key_iv: "30a2ee05a8b89727ef6de7714c5fec30"
server_key_version: "encryptionkey@0"
I20260620 18:36:15.382784 28250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980527431848-22676-0/minicluster-data/master-0/wal/instance:
uuid: "4fc01f70031848de981f25099f8a0355"
format_stamp: "Formatted at 2026-06-20 18:36:15 on dist-test-slave-5lgp"
server_key: "f1f607e75ffc889ccb9bf652734e53d1"
server_key_iv: "30a2ee05a8b89727ef6de7714c5fec30"
server_key_version: "encryptionkey@0"
I20260620 18:36:15.387879 28250 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:36:15.391185 28265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:15.392503 28250 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 18:36:15.392685 28250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980527431848-22676-0/minicluster-data/master-0/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980527431848-22676-0/minicluster-data/master-0/wal
uuid: "4fc01f70031848de981f25099f8a0355"
format_stamp: "Formatted at 2026-06-20 18:36:15 on dist-test-slave-5lgp"
server_key: "f1f607e75ffc889ccb9bf652734e53d1"
server_key_iv: "30a2ee05a8b89727ef6de7714c5fec30"
server_key_version: "encryptionkey@0"
I20260620 18:36:15.392871 28250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980527431848-22676-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980527431848-22676-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980527431848-22676-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:36:15.412433 28250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:36:15.416697 28250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:36:15.416985 28250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:36:15.435492 28250 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.62:41883
I20260620 18:36:15.435528 28316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.62:41883 every 8 connection(s)
I20260620 18:36:15.436858 28250 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
I20260620 18:36:15.438233 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 28250
I20260620 18:36:15.438437 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980527431848-22676-0/minicluster-data/master-0/wal/instance
I20260620 18:36:15.439234 22676 external_mini_cluster.cc:1428] Setting key dbdc2dcd75d6a2b6e1b1dc78596479fb
I20260620 18:36:15.441815 28317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 18:36:15 dist-test-slave-5lgp krb5kdc[28237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980574, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM
I20260620 18:36:15.455060 28317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fc01f70031848de981f25099f8a0355: Bootstrap starting.
I20260620 18:36:15.459394 28317 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4fc01f70031848de981f25099f8a0355: Neither blocks nor log segments found. Creating new log.
I20260620 18:36:15.460743 28317 log.cc:824] T 00000000000000000000000000000000 P 4fc01f70031848de981f25099f8a0355: Log is configured to *not* fsync() on all Append() calls
I20260620 18:36:15.463084 28320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:15.441250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37454 (local address 127.22.37.62:41883)
0620 18:36:15.442106 (+ 856us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:15.442122 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:15.442170 (+ 48us) server_negotiation.cc:407] Connection header received
0620 18:36:15.443779 (+ 1609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:15.443801 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:15.444207 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:15.444548 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:15.446347 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:15.447718 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:15.449513 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:15.450165 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:15.450288 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:15.454993 (+ 4705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:15.455004 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:15.455027 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:15.455088 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:15.458945 (+ 3857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:15.459677 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:15.459688 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:15.459702 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:15.459931 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:15.460289 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:15.460294 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:15.460297 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:15.460998 (+ 701us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:15.461311 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:15.461709 (+ 398us) server_negotiation.cc:299] Negotiation successful
0620 18:36:15.462111 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":658,"thread_start_us":387,"threads_started":1}
I20260620 18:36:15.463824 28317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fc01f70031848de981f25099f8a0355: No bootstrap required, opened a new log
I20260620 18:36:15.471760 28317 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4fc01f70031848de981f25099f8a0355 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fc01f70031848de981f25099f8a0355" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 41883 } }
I20260620 18:36:15.472129 28317 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4fc01f70031848de981f25099f8a0355 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:36:15.472244 28317 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4fc01f70031848de981f25099f8a0355 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fc01f70031848de981f25099f8a0355, State: Initialized, Role: FOLLOWER
I20260620 18:36:15.473052 28317 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4fc01f70031848de981f25099f8a0355 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fc01f70031848de981f25099f8a0355" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 41883 } }
I20260620 18:36:15.473384 28322 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4fc01f70031848de981f25099f8a0355 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:36:15.473631 28322 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4fc01f70031848de981f25099f8a0355 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fc01f70031848de981f25099f8a0355" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 41883 } }
I20260620 18:36:15.473318 28317 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4fc01f70031848de981f25099f8a0355 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:36:15.474337 28317 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4fc01f70031848de981f25099f8a0355 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:36:15.474275 28322 leader_election.cc:302] T 00000000000000000000000000000000 P 4fc01f70031848de981f25099f8a0355 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4fc01f70031848de981f25099f8a0355; no voters:
I20260620 18:36:15.474478 28317 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4fc01f70031848de981f25099f8a0355 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:36:15.475047 28322 leader_election.cc:288] T 00000000000000000000000000000000 P 4fc01f70031848de981f25099f8a0355 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers
I20260620 18:36:15.476873 28317 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4fc01f70031848de981f25099f8a0355 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fc01f70031848de981f25099f8a0355" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 41883 } }
I20260620 18:36:15.477097 28317 leader_election.cc:302] T 00000000000000000000000000000000 P 4fc01f70031848de981f25099f8a0355 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4fc01f70031848de981f25099f8a0355; no voters:
I20260620 18:36:15.477151 28323 raft_consensus.cc:2741] T 00000000000000000000000000000000 P 4fc01f70031848de981f25099f8a0355 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260620 18:36:15.477322 28317 leader_election.cc:288] T 00000000000000000000000000000000 P 4fc01f70031848de981f25099f8a0355 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:36:15.477556 28322 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4fc01f70031848de981f25099f8a0355 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:36:15.478356 28322 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4fc01f70031848de981f25099f8a0355 [term 1 LEADER]: Becoming Leader. State: Replica: 4fc01f70031848de981f25099f8a0355, State: Running, Role: LEADER
I20260620 18:36:15.478816 28322 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4fc01f70031848de981f25099f8a0355 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fc01f70031848de981f25099f8a0355" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 41883 } }
I20260620 18:36:15.479382 28317 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4fc01f70031848de981f25099f8a0355 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:36:15.480988 28324 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fc01f70031848de981f25099f8a0355 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4fc01f70031848de981f25099f8a0355" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fc01f70031848de981f25099f8a0355" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 41883 } } }
I20260620 18:36:15.481508 28324 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fc01f70031848de981f25099f8a0355 [sys.catalog]: This master's current role is: LEADER
I20260620 18:36:15.486591 28323 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fc01f70031848de981f25099f8a0355 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4fc01f70031848de981f25099f8a0355. Latest consensus state: current_term: 1 leader_uuid: "4fc01f70031848de981f25099f8a0355" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fc01f70031848de981f25099f8a0355" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 41883 } } }
I20260620 18:36:15.486863 28332 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:36:15.486932 28323 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fc01f70031848de981f25099f8a0355 [sys.catalog]: This master's current role is: LEADER
I20260620 18:36:15.490676 28332 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:36:15.497776 28332 catalog_manager.cc:1347] Generated new cluster ID: 2b4ad9ae008642ada4fa099f64c6fbf3
I20260620 18:36:15.497871 28332 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:36:15.516304 28332 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:36:15.517627 28332 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:36:15.532240 28332 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4fc01f70031848de981f25099f8a0355: Generated new TSK 0
I20260620 18:36:15.533058 28332 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.37.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.37.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.37.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.1.keytab.
Entry for principal kudu/127.22.37.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.1.keytab.
I20260620 18:36:15.584365 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980527431848-22676-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980527431848-22676-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980527431848-22676-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980527431848-22676-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.37.1:0
--local_ip_for_outbound_sockets=127.22.37.1
--webserver_interface=127.22.37.1
--webserver_port=0
--tserver_master_addrs=127.22.37.62:41883
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.1.keytab
--principal=kudu/127.22.37.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:34181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:36:15.909543 28344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:36:15.909816 28344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:36:15.909955 28344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:36:15.917172 28344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:36:15.917272 28344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:36:15.917462 28344 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.37.1
I20260620 18:36:15.927018 28344 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:34181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980527431848-22676-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980527431848-22676-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.1.keytab
--principal=kudu/127.22.37.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.37.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980527431848-22676-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.37.1
--webserver_port=0
--tserver_master_addrs=127.22.37.62:41883
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.37.1
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980527431848-22676-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:36:15.928323 28344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:36:15.929672 28344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:36:15.942777 28349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:15.942800 28352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:15.942754 28350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:15.943436 28344 server_base.cc:1034] running on GCE node
I20260620 18:36:15.944080 28344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:36:15.945041 28344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:36:15.946262 28344 hybrid_clock.cc:648] HybridClock initialized: now 1781980575946217 us; error 44 us; skew 500 ppm
I20260620 18:36:15.946568 28344 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:36:15 dist-test-slave-5lgp krb5kdc[28237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980575, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:36:15.949630 28344 init.cc:377] Logged in from keytab as kudu/127.22.37.1@KRBTEST.COM (short username kudu)
I20260620 18:36:15.951011 28344 webserver.cc:460] Webserver started at http://127.22.37.1:40009/ using document root <none> and password file <none>
I20260620 18:36:15.951712 28344 fs_manager.cc:359] Metadata directory not provided
I20260620 18:36:15.951862 28344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:36:15.952184 28344 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:36:15.955274 28344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980527431848-22676-0/minicluster-data/ts-0/data/instance:
uuid: "da8939df002d4ad1b0090ce1fe3e147b"
format_stamp: "Formatted at 2026-06-20 18:36:15 on dist-test-slave-5lgp"
server_key: "55b80b784f2d328da5d076b58db20057"
server_key_iv: "f0708d9a5e4d8b85d69f272c03a404cf"
server_key_version: "encryptionkey@0"
I20260620 18:36:15.956070 28344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980527431848-22676-0/minicluster-data/ts-0/wal/instance:
uuid: "da8939df002d4ad1b0090ce1fe3e147b"
format_stamp: "Formatted at 2026-06-20 18:36:15 on dist-test-slave-5lgp"
server_key: "55b80b784f2d328da5d076b58db20057"
server_key_iv: "f0708d9a5e4d8b85d69f272c03a404cf"
server_key_version: "encryptionkey@0"
I20260620 18:36:15.961215 28344 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 18:36:15.965077 28359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:15.966517 28344 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:36:15.966691 28344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980527431848-22676-0/minicluster-data/ts-0/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980527431848-22676-0/minicluster-data/ts-0/wal
uuid: "da8939df002d4ad1b0090ce1fe3e147b"
format_stamp: "Formatted at 2026-06-20 18:36:15 on dist-test-slave-5lgp"
server_key: "55b80b784f2d328da5d076b58db20057"
server_key_iv: "f0708d9a5e4d8b85d69f272c03a404cf"
server_key_version: "encryptionkey@0"
I20260620 18:36:15.966892 28344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980527431848-22676-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980527431848-22676-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980527431848-22676-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:36:15.986402 28344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:36:15.990777 28344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:36:15.991080 28344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:36:15.992019 28344 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:36:15.993347 28344 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:36:15.993503 28344 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:15.993656 28344 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:36:15.993729 28344 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:16.024962 28344 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.1:44693
I20260620 18:36:16.024999 28471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.1:44693 every 8 connection(s)
I20260620 18:36:16.026252 28344 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980527431848-22676-0/minicluster-data/ts-0/data/info.pb
I20260620 18:36:16.036080 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 28344
I20260620 18:36:16.036329 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980527431848-22676-0/minicluster-data/ts-0/wal/instance
I20260620 18:36:16.037070 22676 external_mini_cluster.cc:1428] Setting key 7f922152650718a78ffa5c9fa7982a7d
Jun 20 18:36:16 dist-test-slave-5lgp krb5kdc[28237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980575, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.1@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM
I20260620 18:36:16.045897 28474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:16.028916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.1:49089 (local address 127.22.37.62:41883)
0620 18:36:16.029429 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:16.029434 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:16.030273 (+ 839us) server_negotiation.cc:407] Connection header received
0620 18:36:16.032042 (+ 1769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:16.032048 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:16.032138 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:16.032256 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:16.033728 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:16.034444 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:16.036195 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:16.036662 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:16.036769 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:16.040352 (+ 3583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:16.040357 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:16.040360 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:16.040389 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:16.042948 (+ 2559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:16.043636 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:16.043641 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:16.043652 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:16.043771 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:16.044182 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:16.044187 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:16.044190 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:16.044459 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:16.044603 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:16.045264 (+ 661us) server_negotiation.cc:299] Negotiation successful
0620 18:36:16.045504 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":243,"threads_started":1}
I20260620 18:36:16.046842 28475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:16.029362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:41883 (local address 127.22.37.1:49089)
0620 18:36:16.030104 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:16.030149 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:16.031780 (+ 1631us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:16.032522 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:16.032530 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:16.033018 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:16.033561 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:16.033577 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:16.034590 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:16.034603 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:16.036025 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:16.036043 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:16.036896 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:16.036904 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:16.037202 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:16.038039 (+ 837us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:16.038064 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:16.040136 (+ 2072us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:36:16.043084 (+ 2948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:16.043093 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:16.043108 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:16.043512 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:16.043877 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:16.043883 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:16.043887 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:16.044066 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:16.044590 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:36:16.044598 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:36:16.044920 (+ 322us) client_negotiation.cc:769] Sending connection context
0620 18:36:16.045176 (+ 256us) client_negotiation.cc:240] Negotiation successful
0620 18:36:16.045734 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":276,"threads_started":1}
I20260620 18:36:16.048349 28472 heartbeater.cc:347] Connected to a master server at 127.22.37.62:41883
I20260620 18:36:16.048645 28472 heartbeater.cc:464] Registering TS with master...
I20260620 18:36:16.049337 28472 heartbeater.cc:511] Master 127.22.37.62:41883 requested a full tablet report, sending...
I20260620 18:36:16.051682 28282 ts_manager.cc:194] Registered new tserver with Master: da8939df002d4ad1b0090ce1fe3e147b (127.22.37.1:44693)
I20260620 18:36:16.053921 28282 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.37.1@KRBTEST.COM'} at 127.22.37.1:49089
WARNING: no policy specified for kudu/127.22.37.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.37.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.37.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.2.keytab.
Entry for principal kudu/127.22.37.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.2.keytab.
I20260620 18:36:16.080886 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980527431848-22676-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980527431848-22676-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980527431848-22676-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980527431848-22676-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.37.2:0
--local_ip_for_outbound_sockets=127.22.37.2
--webserver_interface=127.22.37.2
--webserver_port=0
--tserver_master_addrs=127.22.37.62:41883
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.2.keytab
--principal=kudu/127.22.37.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:34181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:36:16.401242 28478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:36:16.401659 28478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:36:16.401850 28478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:36:16.409162 28478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:36:16.409322 28478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:36:16.409511 28478 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.37.2
I20260620 18:36:16.419718 28478 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:34181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980527431848-22676-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980527431848-22676-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.2.keytab
--principal=kudu/127.22.37.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.37.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980527431848-22676-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.37.2
--webserver_port=0
--tserver_master_addrs=127.22.37.62:41883
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.37.2
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980527431848-22676-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:36:16.421085 28478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:36:16.422513 28478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:36:16.436178 28483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:16.436133 28484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:16.436798 28486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:16.437486 28478 server_base.cc:1034] running on GCE node
I20260620 18:36:16.438109 28478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:36:16.439064 28478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:36:16.440299 28478 hybrid_clock.cc:648] HybridClock initialized: now 1781980576440249 us; error 41 us; skew 500 ppm
I20260620 18:36:16.440620 28478 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:36:16 dist-test-slave-5lgp krb5kdc[28237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980576, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:36:16.443966 28478 init.cc:377] Logged in from keytab as kudu/127.22.37.2@KRBTEST.COM (short username kudu)
I20260620 18:36:16.449923 28478 webserver.cc:460] Webserver started at http://127.22.37.2:44929/ using document root <none> and password file <none>
I20260620 18:36:16.450636 28478 fs_manager.cc:359] Metadata directory not provided
I20260620 18:36:16.450783 28478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:36:16.451090 28478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:36:16.454066 28478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980527431848-22676-0/minicluster-data/ts-1/data/instance:
uuid: "7abbbefaf6ed4550baf52e241deb651e"
format_stamp: "Formatted at 2026-06-20 18:36:16 on dist-test-slave-5lgp"
server_key: "4007bc7a5b8f5e0d9c64d39bff1172a0"
server_key_iv: "8dcbe0c8f5658671c3f14c6b2212a05d"
server_key_version: "encryptionkey@0"
I20260620 18:36:16.454825 28478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980527431848-22676-0/minicluster-data/ts-1/wal/instance:
uuid: "7abbbefaf6ed4550baf52e241deb651e"
format_stamp: "Formatted at 2026-06-20 18:36:16 on dist-test-slave-5lgp"
server_key: "4007bc7a5b8f5e0d9c64d39bff1172a0"
server_key_iv: "8dcbe0c8f5658671c3f14c6b2212a05d"
server_key_version: "encryptionkey@0"
I20260620 18:36:16.459847 28478 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260620 18:36:16.463166 28493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:16.464470 28478 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 18:36:16.464637 28478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980527431848-22676-0/minicluster-data/ts-1/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980527431848-22676-0/minicluster-data/ts-1/wal
uuid: "7abbbefaf6ed4550baf52e241deb651e"
format_stamp: "Formatted at 2026-06-20 18:36:16 on dist-test-slave-5lgp"
server_key: "4007bc7a5b8f5e0d9c64d39bff1172a0"
server_key_iv: "8dcbe0c8f5658671c3f14c6b2212a05d"
server_key_version: "encryptionkey@0"
I20260620 18:36:16.464833 28478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980527431848-22676-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980527431848-22676-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980527431848-22676-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:36:16.479827 28478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:36:16.484215 28478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:36:16.484510 28478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:36:16.485523 28478 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:36:16.486774 28478 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:36:16.486883 28478 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:16.487026 28478 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:36:16.487100 28478 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:16.519726 28478 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.2:36999
I20260620 18:36:16.519778 28605 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.2:36999 every 8 connection(s)
I20260620 18:36:16.521061 28478 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980527431848-22676-0/minicluster-data/ts-1/data/info.pb
I20260620 18:36:16.530438 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 28478
I20260620 18:36:16.530716 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980527431848-22676-0/minicluster-data/ts-1/wal/instance
I20260620 18:36:16.531183 22676 external_mini_cluster.cc:1428] Setting key 6a2d965071a57427b64ef9b1d53b588a
Jun 20 18:36:16 dist-test-slave-5lgp krb5kdc[28237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980576, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.2@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM
I20260620 18:36:16.540472 28474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:16.523621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.2:59911 (local address 127.22.37.62:41883)
0620 18:36:16.523832 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:16.523838 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:16.525008 (+ 1170us) server_negotiation.cc:407] Connection header received
0620 18:36:16.526781 (+ 1773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:16.526786 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:16.526853 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:16.526941 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:16.528408 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:16.529127 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:16.530727 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:16.531187 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:16.531334 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:16.535156 (+ 3822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:16.535161 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:16.535164 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:16.535193 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:16.537608 (+ 2415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:16.538291 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:16.538295 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:16.538297 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:16.538414 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:16.538839 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:16.538850 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:16.538852 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:16.539093 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:16.539243 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:16.539905 (+ 662us) server_negotiation.cc:299] Negotiation successful
0620 18:36:16.540127 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 18:36:16.541491 28608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:16.524135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:41883 (local address 127.22.37.2:59911)
0620 18:36:16.524846 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:16.524883 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:16.526527 (+ 1644us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:16.527175 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:16.527183 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:16.527661 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:16.528258 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:16.528274 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:16.529260 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:16.529264 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:16.530556 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:16.530573 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:16.531418 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:16.531427 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:16.531694 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:16.532511 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:16.532539 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:16.534942 (+ 2403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:36:16.537733 (+ 2791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:16.537751 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:16.537774 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:16.538157 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:16.538530 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:16.538536 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:16.538540 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:16.538719 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:16.539215 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:36:16.539222 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:36:16.539554 (+ 332us) client_negotiation.cc:769] Sending connection context
0620 18:36:16.539823 (+ 269us) client_negotiation.cc:240] Negotiation successful
0620 18:36:16.540323 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":274,"threads_started":1}
I20260620 18:36:16.542958 28606 heartbeater.cc:347] Connected to a master server at 127.22.37.62:41883
I20260620 18:36:16.543285 28606 heartbeater.cc:464] Registering TS with master...
I20260620 18:36:16.544121 28606 heartbeater.cc:511] Master 127.22.37.62:41883 requested a full tablet report, sending...
I20260620 18:36:16.545822 28282 ts_manager.cc:194] Registered new tserver with Master: 7abbbefaf6ed4550baf52e241deb651e (127.22.37.2:36999)
I20260620 18:36:16.547099 28282 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.37.2@KRBTEST.COM'} at 127.22.37.2:59911
WARNING: no policy specified for kudu/127.22.37.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.37.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.37.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.3.keytab.
Entry for principal kudu/127.22.37.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.3.keytab.
I20260620 18:36:16.576932 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980527431848-22676-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980527431848-22676-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980527431848-22676-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980527431848-22676-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.37.3:0
--local_ip_for_outbound_sockets=127.22.37.3
--webserver_interface=127.22.37.3
--webserver_port=0
--tserver_master_addrs=127.22.37.62:41883
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.3.keytab
--principal=kudu/127.22.37.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:34181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:36:16.900722 28611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:36:16.901014 28611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:36:16.901108 28611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:36:16.908331 28611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:36:16.908434 28611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:36:16.908546 28611 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.37.3
I20260620 18:36:16.918387 28611 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:34181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980527431848-22676-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980527431848-22676-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.3.keytab
--principal=kudu/127.22.37.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.37.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980527431848-22676-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.37.3
--webserver_port=0
--tserver_master_addrs=127.22.37.62:41883
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.37.3
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980527431848-22676-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:36:16.919688 28611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:36:16.920960 28611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:36:16.934914 28617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:16.934865 28616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:16.934944 28619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:16.935667 28611 server_base.cc:1034] running on GCE node
I20260620 18:36:16.936272 28611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:36:16.937181 28611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:36:16.938387 28611 hybrid_clock.cc:648] HybridClock initialized: now 1781980576938333 us; error 48 us; skew 500 ppm
I20260620 18:36:16.938740 28611 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:36:16 dist-test-slave-5lgp krb5kdc[28237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980576, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:36:16.941938 28611 init.cc:377] Logged in from keytab as kudu/127.22.37.3@KRBTEST.COM (short username kudu)
I20260620 18:36:16.943318 28611 webserver.cc:460] Webserver started at http://127.22.37.3:33211/ using document root <none> and password file <none>
I20260620 18:36:16.944006 28611 fs_manager.cc:359] Metadata directory not provided
I20260620 18:36:16.944131 28611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:36:16.944454 28611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:36:16.947443 28611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980527431848-22676-0/minicluster-data/ts-2/data/instance:
uuid: "8b768bcc5fc7410cb2d7a3362d669b04"
format_stamp: "Formatted at 2026-06-20 18:36:16 on dist-test-slave-5lgp"
server_key: "3c98912de1ea254b19122f46b99046ce"
server_key_iv: "9a9857d69647ab518fec76e5debbf306"
server_key_version: "encryptionkey@0"
I20260620 18:36:16.948211 28611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980527431848-22676-0/minicluster-data/ts-2/wal/instance:
uuid: "8b768bcc5fc7410cb2d7a3362d669b04"
format_stamp: "Formatted at 2026-06-20 18:36:16 on dist-test-slave-5lgp"
server_key: "3c98912de1ea254b19122f46b99046ce"
server_key_iv: "9a9857d69647ab518fec76e5debbf306"
server_key_version: "encryptionkey@0"
I20260620 18:36:16.953562 28611 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.006s
I20260620 18:36:16.957144 28626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:16.958556 28611 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:36:16.958722 28611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980527431848-22676-0/minicluster-data/ts-2/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980527431848-22676-0/minicluster-data/ts-2/wal
uuid: "8b768bcc5fc7410cb2d7a3362d669b04"
format_stamp: "Formatted at 2026-06-20 18:36:16 on dist-test-slave-5lgp"
server_key: "3c98912de1ea254b19122f46b99046ce"
server_key_iv: "9a9857d69647ab518fec76e5debbf306"
server_key_version: "encryptionkey@0"
I20260620 18:36:16.958945 28611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980527431848-22676-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980527431848-22676-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980527431848-22676-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:36:16.981333 28611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:36:16.986106 28611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:36:16.986434 28611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:36:16.987427 28611 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:36:16.988804 28611 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:36:16.988878 28611 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:16.989050 28611 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:36:16.989099 28611 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:17.022759 28611 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.3:42603
I20260620 18:36:17.022868 28738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.3:42603 every 8 connection(s)
I20260620 18:36:17.024096 28611 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980527431848-22676-0/minicluster-data/ts-2/data/info.pb
I20260620 18:36:17.026397 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 28611
I20260620 18:36:17.026580 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980527431848-22676-0/minicluster-data/ts-2/wal/instance
I20260620 18:36:17.027159 22676 external_mini_cluster.cc:1428] Setting key 16b2bb07cbc00f613338056c93ba6ce4
Jun 20 18:36:17 dist-test-slave-5lgp krb5kdc[28237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980576, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.3@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM
I20260620 18:36:17.045275 28474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:17.027091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.3:44685 (local address 127.22.37.62:41883)
0620 18:36:17.027290 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:17.027296 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:17.028694 (+ 1398us) server_negotiation.cc:407] Connection header received
0620 18:36:17.031183 (+ 2489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:17.031187 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:17.031253 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:17.031339 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:17.032786 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:17.033594 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:17.035081 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:17.035488 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:17.035584 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:17.039025 (+ 3441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:17.039030 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:17.039033 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:17.039063 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:17.042055 (+ 2992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:17.042948 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:17.042955 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:17.042959 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:17.043096 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:17.043528 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:17.043534 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:17.043538 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:17.043821 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:17.043971 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:17.044670 (+ 699us) server_negotiation.cc:299] Negotiation successful
0620 18:36:17.044898 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 18:36:17.046291 28741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:17.027670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:41883 (local address 127.22.37.3:44685)
0620 18:36:17.028542 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:17.028588 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:17.030898 (+ 2310us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:17.031592 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:17.031601 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:17.032088 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:17.032629 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:17.032646 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:17.033735 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:17.033738 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:17.034954 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:17.034967 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:17.035667 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:17.035671 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:17.035886 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:17.036601 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:17.036623 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:17.038819 (+ 2196us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:36:17.042180 (+ 3361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:17.042189 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:17.042208 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:17.042822 (+ 614us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:17.043209 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:17.043214 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:17.043218 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:17.043416 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:17.043949 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:36:17.043957 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:36:17.044325 (+ 368us) client_negotiation.cc:769] Sending connection context
0620 18:36:17.044599 (+ 274us) client_negotiation.cc:240] Negotiation successful
0620 18:36:17.045141 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":328,"threads_started":1}
I20260620 18:36:17.047688 28739 heartbeater.cc:347] Connected to a master server at 127.22.37.62:41883
I20260620 18:36:17.048026 28739 heartbeater.cc:464] Registering TS with master...
I20260620 18:36:17.048761 28739 heartbeater.cc:511] Master 127.22.37.62:41883 requested a full tablet report, sending...
I20260620 18:36:17.050249 28282 ts_manager.cc:194] Registered new tserver with Master: 8b768bcc5fc7410cb2d7a3362d669b04 (127.22.37.3:42603)
I20260620 18:36:17.051105 28282 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.37.3@KRBTEST.COM'} at 127.22.37.3:44685
I20260620 18:36:17.055259 22676 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:36:17.057125 28472 heartbeater.cc:503] Master 127.22.37.62:41883 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980527431848-22676-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980527431848-22676-0/krb5kdc/test-admin.keytab.
Jun 20 18:36:17 dist-test-slave-5lgp krb5kdc[28237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:36:17.082957 22676 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260620 18:36:17.088059 28746 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:36:17.085649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48812 (local address 127.22.37.1:44693)
0620 18:36:17.086327 (+ 678us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:17.086335 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:17.086362 (+ 27us) server_negotiation.cc:407] Connection header received
0620 18:36:17.086421 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:17.086428 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:17.086646 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:17.086776 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:17.087616 (+ 840us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48812: BlockingRecv error: recv got EOF from 127.0.0.1:48812 (error 108)
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":334,"threads_started":1}
W20260620 18:36:17.088093 22676 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980527431848-22676-0/krb5kdc/krb5cc' of type FILE
Jun 20 18:36:17 dist-test-slave-5lgp krb5kdc[28237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:36:17.089728 22676 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 18:36:17 dist-test-slave-5lgp krb5kdc[28237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:36:17.091167 22676 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 18:36:17 dist-test-slave-5lgp krb5kdc[28237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.37.1@KRBTEST.COM
I20260620 18:36:17.106372 28746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:17.092759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48820 (local address 127.22.37.1:44693)
0620 18:36:17.092966 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:17.092974 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:17.093238 (+ 264us) server_negotiation.cc:407] Connection header received
0620 18:36:17.093465 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:17.093468 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:17.093528 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:17.093637 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:17.094607 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:17.095573 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:17.096902 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:17.097584 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:17.097718 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:17.101190 (+ 3472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:17.101201 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:17.101215 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:17.101246 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:17.103968 (+ 2722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:17.104565 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:17.104573 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:17.104575 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:17.104688 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:17.104997 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:17.105000 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:17.105002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:17.105330 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:17.105646 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:17.105897 (+ 251us) server_negotiation.cc:299] Negotiation successful
0620 18:36:17.106085 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260620 18:36:17.112085 28746 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:36:17.110376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48826 (local address 127.22.37.1:44693)
0620 18:36:17.110584 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:17.110592 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:17.110812 (+ 220us) server_negotiation.cc:407] Connection header received
0620 18:36:17.110983 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:17.110988 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:17.111056 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:17.111150 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:17.111849 (+ 699us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48826: BlockingRecv error: recv got EOF from 127.0.0.1:48826 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260620 18:36:17.112476 22676 init.cc:244] Error while opening credential cache '/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980527431848-22676-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980527431848-22676-0/krb5kdc/krb5cc)
Jun 20 18:36:17 dist-test-slave-5lgp krb5kdc[28237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:36:17.113960 22676 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 18:36:17 dist-test-slave-5lgp krb5kdc[28237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:36:17.115181 22676 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 18:36:17 dist-test-slave-5lgp krb5kdc[28237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.37.1@KRBTEST.COM
I20260620 18:36:17.128427 28746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:17.116557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48840 (local address 127.22.37.1:44693)
0620 18:36:17.116750 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:17.116755 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:17.116833 (+ 78us) server_negotiation.cc:407] Connection header received
0620 18:36:17.116945 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:17.116948 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:17.117001 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:17.117078 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:17.118036 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:17.118731 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:17.119868 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:17.120276 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:17.120376 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:17.123692 (+ 3316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:17.123697 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:17.123700 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:17.123730 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:17.126236 (+ 2506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:17.126779 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:17.126783 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:17.126785 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:17.126898 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:17.127275 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:17.127278 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:17.127280 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:17.127529 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:17.127723 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:17.127943 (+ 220us) server_negotiation.cc:299] Negotiation successful
0620 18:36:17.128085 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260620 18:36:17.132858 28746 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:36:17.131127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48848 (local address 127.22.37.1:44693)
0620 18:36:17.131329 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:17.131338 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:17.131611 (+ 273us) server_negotiation.cc:407] Connection header received
0620 18:36:17.131779 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:17.131784 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:17.131859 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:17.131956 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:17.132608 (+ 652us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48848: BlockingRecv error: recv got EOF from 127.0.0.1:48848 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260620 18:36:17.132992 22676 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980527431848-22676-0/krb5kdc/krb5cc' of type FILE
Jun 20 18:36:17 dist-test-slave-5lgp krb5kdc[28237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:36:17.134521 22676 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 18:36:17 dist-test-slave-5lgp krb5kdc[28237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:36:17.135720 22676 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 18:36:17 dist-test-slave-5lgp krb5kdc[28237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.37.1@KRBTEST.COM
I20260620 18:36:17.155339 28746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:17.137156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48852 (local address 127.22.37.1:44693)
0620 18:36:17.137406 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:17.137412 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:17.142053 (+ 4641us) server_negotiation.cc:407] Connection header received
0620 18:36:17.142242 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:17.142247 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:17.142344 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:17.142464 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:17.143793 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:17.144936 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:17.146690 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:17.147241 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:17.147374 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:17.150787 (+ 3413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:17.150792 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:17.150795 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:17.150823 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:17.153146 (+ 2323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:17.153797 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:17.153800 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:17.153803 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:17.153955 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:17.154275 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:17.154278 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:17.154280 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:17.154509 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:17.154727 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:17.154918 (+ 191us) server_negotiation.cc:299] Negotiation successful
0620 18:36:17.155071 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 18:36:17.157027 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 28344
I20260620 18:36:17.168001 28746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:17.157760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48856 (local address 127.22.37.1:44693)
0620 18:36:17.157975 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:17.157984 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:17.158272 (+ 288us) server_negotiation.cc:407] Connection header received
0620 18:36:17.158368 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:17.158371 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:17.158422 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:17.158494 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:17.159301 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:17.159932 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:17.161006 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:17.161508 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:17.161597 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:17.163519 (+ 1922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:17.163523 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:17.163527 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:17.163552 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:17.165935 (+ 2383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:17.166477 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:17.166480 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:17.166483 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:17.166602 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:17.166939 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:17.166942 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:17.166944 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:17.167154 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:17.167335 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:17.167552 (+ 217us) server_negotiation.cc:299] Negotiation successful
0620 18:36:17.167764 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 18:36:17.168599 28467 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:36:17.250070 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 28344
I20260620 18:36:17.266646 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 28478
Jun 20 18:36:17 dist-test-slave-5lgp krb5kdc[28237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.37.2@KRBTEST.COM
I20260620 18:36:17.283056 28759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:17.267547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55258 (local address 127.22.37.2:36999)
0620 18:36:17.268116 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:17.268123 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:17.268150 (+ 27us) server_negotiation.cc:407] Connection header received
0620 18:36:17.268214 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:17.268220 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:17.268434 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:17.268645 (+ 211us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:17.269846 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:17.270916 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:17.272212 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:17.272796 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:17.272968 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:17.276890 (+ 3922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:17.276901 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:17.276915 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:17.276947 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:17.279845 (+ 2898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:17.280661 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:17.280668 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:17.280670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:17.280801 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:17.281267 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:17.281273 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:17.281277 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:17.281752 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:17.282093 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:17.282467 (+ 374us) server_negotiation.cc:299] Negotiation successful
0620 18:36:17.282706 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":256,"threads_started":1}
I20260620 18:36:17.284360 28601 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:36:17.366392 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 28478
I20260620 18:36:17.382181 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 28611
Jun 20 18:36:17 dist-test-slave-5lgp krb5kdc[28237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.37.3@KRBTEST.COM
I20260620 18:36:17.397737 28761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:17.383283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39488 (local address 127.22.37.3:42603)
0620 18:36:17.383981 (+ 698us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:17.383989 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:17.384016 (+ 27us) server_negotiation.cc:407] Connection header received
0620 18:36:17.384075 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:17.384081 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:17.384303 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:17.384430 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:17.385485 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:17.386477 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:17.387636 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:17.388204 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:17.388297 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:17.392256 (+ 3959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:17.392266 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:17.392281 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:17.392318 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:17.395137 (+ 2819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:17.395830 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:17.395837 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:17.395839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:17.395950 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:17.396290 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:17.396294 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:17.396295 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:17.396618 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:17.396842 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:17.397098 (+ 256us) server_negotiation.cc:299] Negotiation successful
0620 18:36:17.397428 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":382,"threads_started":1}
I20260620 18:36:17.399000 28734 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:36:17.481011 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 28611
I20260620 18:36:17.496733 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 28250
I20260620 18:36:17.497967 28312 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:36:17.599761 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 28250
2026-06-20T18:36:17Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3872 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:36:17 dist-test-slave-5lgp krb5kdc[28766](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:36:17 dist-test-slave-5lgp krb5kdc[28766](info): set up 2 sockets
Jun 20 18:36:17 dist-test-slave-5lgp krb5kdc[28766](info): commencing operation
krb5kdc: starting...
W20260620 18:36:18.698246 22676 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.003s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/krb5kdc/test-user.keytab.
Jun 20 18:36:18 dist-test-slave-5lgp krb5kdc[28766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980578, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:36:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:36:18Z Disabled control of system clock
WARNING: no policy specified for oryx/127.22.37.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.37.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.37.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/krb5kdc/oryx_127.22.37.62.keytab.
Entry for principal oryx/127.22.37.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/krb5kdc/oryx_127.22.37.62.keytab.
I20260620 18:36:18.900179 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.62:46099
--webserver_interface=127.22.37.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/krb5kdc/oryx_127.22.37.62.keytab
--principal=oryx/127.22.37.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:38175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.37.62:46099
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:36:19.220137 28779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:36:19.220413 28779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:36:19.220584 28779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:36:19.227686 28779 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:36:19.227787 28779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:36:19.227888 28779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:36:19.227991 28779 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:36:19.228075 28779 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:36:19.237845 28779 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:38175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.37.62:46099
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/krb5kdc/oryx_127.22.37.62.keytab
--principal=oryx/127.22.37.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.62:46099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.37.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:36:19.239125 28779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:36:19.240476 28779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:36:19.251734 28787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:19.251827 28784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:19.251861 28785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:19.252772 28779 server_base.cc:1034] running on GCE node
I20260620 18:36:19.253629 28779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:36:19.254900 28779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:36:19.256124 28779 hybrid_clock.cc:648] HybridClock initialized: now 1781980579256075 us; error 42 us; skew 500 ppm
I20260620 18:36:19.256474 28779 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:36:19 dist-test-slave-5lgp krb5kdc[28766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980579, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.37.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:36:19.263247 28779 init.cc:377] Logged in from keytab as oryx/127.22.37.62@KRBTEST.COM (short username oryx)
I20260620 18:36:19.264760 28779 webserver.cc:460] Webserver started at http://127.22.37.62:37711/ using document root <none> and password file <none>
I20260620 18:36:19.265532 28779 fs_manager.cc:359] Metadata directory not provided
I20260620 18:36:19.265677 28779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:36:19.265982 28779 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:36:19.268869 28779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/master-0/data/instance:
uuid: "c8fe34680f3c4d388068736f074cb839"
format_stamp: "Formatted at 2026-06-20 18:36:19 on dist-test-slave-5lgp"
server_key: "f1783e802174c2309f5166f1f37a857f"
server_key_iv: "e0d7824af04818041a95841fb87bb2fa"
server_key_version: "encryptionkey@0"
I20260620 18:36:19.269642 28779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/master-0/wal/instance:
uuid: "c8fe34680f3c4d388068736f074cb839"
format_stamp: "Formatted at 2026-06-20 18:36:19 on dist-test-slave-5lgp"
server_key: "f1783e802174c2309f5166f1f37a857f"
server_key_iv: "e0d7824af04818041a95841fb87bb2fa"
server_key_version: "encryptionkey@0"
I20260620 18:36:19.274472 28779 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 18:36:19.277873 28794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:19.279110 28779 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 18:36:19.279291 28779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/master-0/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/master-0/wal
uuid: "c8fe34680f3c4d388068736f074cb839"
format_stamp: "Formatted at 2026-06-20 18:36:19 on dist-test-slave-5lgp"
server_key: "f1783e802174c2309f5166f1f37a857f"
server_key_iv: "e0d7824af04818041a95841fb87bb2fa"
server_key_version: "encryptionkey@0"
I20260620 18:36:19.279490 28779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:36:19.322645 28779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:36:19.327174 28779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:36:19.327486 28779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:36:19.345745 28779 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.62:46099
I20260620 18:36:19.345777 28845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.62:46099 every 8 connection(s)
I20260620 18:36:19.347100 28779 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
I20260620 18:36:19.349134 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 28779
I20260620 18:36:19.349359 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/master-0/wal/instance
I20260620 18:36:19.350035 22676 external_mini_cluster.cc:1428] Setting key db5214aa0b5ee81ab57b4cdbd950af55
I20260620 18:36:19.351166 28846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 18:36:19 dist-test-slave-5lgp krb5kdc[28766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980578, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM
I20260620 18:36:19.364773 28846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8fe34680f3c4d388068736f074cb839: Bootstrap starting.
I20260620 18:36:19.368860 28846 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c8fe34680f3c4d388068736f074cb839: Neither blocks nor log segments found. Creating new log.
I20260620 18:36:19.370149 28846 log.cc:824] T 00000000000000000000000000000000 P c8fe34680f3c4d388068736f074cb839: Log is configured to *not* fsync() on all Append() calls
I20260620 18:36:19.373478 28846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8fe34680f3c4d388068736f074cb839: No bootstrap required, opened a new log
I20260620 18:36:19.373762 28849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:19.352023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43802 (local address 127.22.37.62:46099)
0620 18:36:19.352845 (+ 822us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:19.352860 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:19.352909 (+ 49us) server_negotiation.cc:407] Connection header received
0620 18:36:19.354697 (+ 1788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:19.354726 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:19.355250 (+ 524us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:19.355693 (+ 443us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:19.356961 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:19.359060 (+ 2099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:19.360328 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:19.361418 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:19.361608 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:19.365638 (+ 4030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:19.365648 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:19.365665 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:19.365712 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:19.369504 (+ 3792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:19.370147 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:19.370155 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:19.370164 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:19.370352 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:19.370728 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:19.370736 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:19.370742 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:19.371318 (+ 576us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:19.371623 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:19.371992 (+ 369us) server_negotiation.cc:299] Negotiation successful
0620 18:36:19.372399 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":577,"thread_start_us":335,"threads_started":1}
I20260620 18:36:19.381340 28846 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8fe34680f3c4d388068736f074cb839 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8fe34680f3c4d388068736f074cb839" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 46099 } }
I20260620 18:36:19.381822 28846 raft_consensus.cc:374] T 00000000000000000000000000000000 P c8fe34680f3c4d388068736f074cb839 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:36:19.381969 28846 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8fe34680f3c4d388068736f074cb839 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8fe34680f3c4d388068736f074cb839, State: Initialized, Role: FOLLOWER
I20260620 18:36:19.382576 28846 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8fe34680f3c4d388068736f074cb839 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8fe34680f3c4d388068736f074cb839" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 46099 } }
I20260620 18:36:19.382818 28846 raft_consensus.cc:388] T 00000000000000000000000000000000 P c8fe34680f3c4d388068736f074cb839 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:36:19.382962 28846 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8fe34680f3c4d388068736f074cb839 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:36:19.383145 28846 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8fe34680f3c4d388068736f074cb839 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:36:19.385325 28846 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8fe34680f3c4d388068736f074cb839 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8fe34680f3c4d388068736f074cb839" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 46099 } }
I20260620 18:36:19.385980 28846 leader_election.cc:302] T 00000000000000000000000000000000 P c8fe34680f3c4d388068736f074cb839 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c8fe34680f3c4d388068736f074cb839; no voters:
I20260620 18:36:19.386610 28846 leader_election.cc:288] T 00000000000000000000000000000000 P c8fe34680f3c4d388068736f074cb839 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:36:19.386732 28851 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8fe34680f3c4d388068736f074cb839 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:36:19.387444 28851 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8fe34680f3c4d388068736f074cb839 [term 1 LEADER]: Becoming Leader. State: Replica: c8fe34680f3c4d388068736f074cb839, State: Running, Role: LEADER
I20260620 18:36:19.387930 28851 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8fe34680f3c4d388068736f074cb839 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8fe34680f3c4d388068736f074cb839" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 46099 } }
I20260620 18:36:19.388720 28846 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8fe34680f3c4d388068736f074cb839 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:36:19.390625 28853 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8fe34680f3c4d388068736f074cb839 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8fe34680f3c4d388068736f074cb839. Latest consensus state: current_term: 1 leader_uuid: "c8fe34680f3c4d388068736f074cb839" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8fe34680f3c4d388068736f074cb839" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 46099 } } }
I20260620 18:36:19.391041 28853 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8fe34680f3c4d388068736f074cb839 [sys.catalog]: This master's current role is: LEADER
I20260620 18:36:19.393189 28852 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8fe34680f3c4d388068736f074cb839 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c8fe34680f3c4d388068736f074cb839" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8fe34680f3c4d388068736f074cb839" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 46099 } } }
I20260620 18:36:19.393497 28857 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:36:19.393644 28852 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8fe34680f3c4d388068736f074cb839 [sys.catalog]: This master's current role is: LEADER
I20260620 18:36:19.401265 28857 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:36:19.409989 28857 catalog_manager.cc:1347] Generated new cluster ID: eee720b350864e718f512e0ff17d1a2e
I20260620 18:36:19.410162 28857 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:36:19.425849 28857 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:36:19.427201 28857 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:36:19.439116 28857 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c8fe34680f3c4d388068736f074cb839: Generated new TSK 0
I20260620 18:36:19.440014 28857 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.22.37.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.37.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.37.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/krb5kdc/oryx_127.22.37.1.keytab.
Entry for principal oryx/127.22.37.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/krb5kdc/oryx_127.22.37.1.keytab.
I20260620 18:36:19.488682 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.37.1:0
--local_ip_for_outbound_sockets=127.22.37.1
--webserver_interface=127.22.37.1
--webserver_port=0
--tserver_master_addrs=127.22.37.62:46099
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/krb5kdc/oryx_127.22.37.1.keytab
--principal=oryx/127.22.37.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:38175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:36:19.821076 28872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:36:19.821414 28872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:36:19.821537 28872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:36:19.829051 28872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:36:19.829159 28872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:36:19.829281 28872 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.37.1
I20260620 18:36:19.839200 28872 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:38175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/krb5kdc/oryx_127.22.37.1.keytab
--principal=oryx/127.22.37.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.37.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.37.1
--webserver_port=0
--tserver_master_addrs=127.22.37.62:46099
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.37.1
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:36:19.840416 28872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:36:19.841688 28872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:36:19.854872 28877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:19.854846 28878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:19.854936 28880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:19.855587 28872 server_base.cc:1034] running on GCE node
I20260620 18:36:19.856208 28872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:36:19.857152 28872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:36:19.858356 28872 hybrid_clock.cc:648] HybridClock initialized: now 1781980579858277 us; error 68 us; skew 500 ppm
I20260620 18:36:19.858677 28872 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:36:19 dist-test-slave-5lgp krb5kdc[28766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980579, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.37.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:36:19.861690 28872 init.cc:377] Logged in from keytab as oryx/127.22.37.1@KRBTEST.COM (short username oryx)
I20260620 18:36:19.863081 28872 webserver.cc:460] Webserver started at http://127.22.37.1:42569/ using document root <none> and password file <none>
I20260620 18:36:19.863780 28872 fs_manager.cc:359] Metadata directory not provided
I20260620 18:36:19.863933 28872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:36:19.864267 28872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:36:19.867238 28872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/ts-0/data/instance:
uuid: "7232cd23c889418689f2c886ba008313"
format_stamp: "Formatted at 2026-06-20 18:36:19 on dist-test-slave-5lgp"
server_key: "c7e09bd1cd6e2bbc61456e599776464e"
server_key_iv: "700bfd76d7261ab63464ff09a27da685"
server_key_version: "encryptionkey@0"
I20260620 18:36:19.868032 28872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/ts-0/wal/instance:
uuid: "7232cd23c889418689f2c886ba008313"
format_stamp: "Formatted at 2026-06-20 18:36:19 on dist-test-slave-5lgp"
server_key: "c7e09bd1cd6e2bbc61456e599776464e"
server_key_iv: "700bfd76d7261ab63464ff09a27da685"
server_key_version: "encryptionkey@0"
I20260620 18:36:19.873484 28872 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260620 18:36:19.877717 28887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:19.879055 28872 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:36:19.879220 28872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/ts-0/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/ts-0/wal
uuid: "7232cd23c889418689f2c886ba008313"
format_stamp: "Formatted at 2026-06-20 18:36:19 on dist-test-slave-5lgp"
server_key: "c7e09bd1cd6e2bbc61456e599776464e"
server_key_iv: "700bfd76d7261ab63464ff09a27da685"
server_key_version: "encryptionkey@0"
I20260620 18:36:19.879428 28872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:36:19.897519 28872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:36:19.901193 28872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:36:19.901489 28872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:36:19.902426 28872 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:36:19.903721 28872 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:36:19.903791 28872 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:19.903962 28872 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:36:19.904036 28872 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:19.936684 28872 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.1:35961
I20260620 18:36:19.936725 28999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.1:35961 every 8 connection(s)
I20260620 18:36:19.938058 28872 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/ts-0/data/info.pb
I20260620 18:36:19.939306 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 28872
I20260620 18:36:19.939509 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/ts-0/wal/instance
I20260620 18:36:19.940340 22676 external_mini_cluster.cc:1428] Setting key edcab1fbe74401964b6f4473bd5c6c64
Jun 20 18:36:19 dist-test-slave-5lgp krb5kdc[28766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980579, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.37.1@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM
I20260620 18:36:19.961854 29002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:19.941165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.1:47877 (local address 127.22.37.62:46099)
0620 18:36:19.941669 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:19.941674 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:19.942810 (+ 1136us) server_negotiation.cc:407] Connection header received
0620 18:36:19.945727 (+ 2917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:19.945732 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:19.945795 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:19.945884 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:19.948091 (+ 2207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:19.948793 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:19.950614 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:19.951368 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:19.951516 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:19.955043 (+ 3527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:19.955052 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:19.955060 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:19.955104 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:19.958642 (+ 3538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:19.959355 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:19.959364 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:19.959370 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:19.959522 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:19.959925 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:19.959932 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:19.959937 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:19.960366 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:19.960516 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:19.961155 (+ 639us) server_negotiation.cc:299] Negotiation successful
0620 18:36:19.961440 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":303,"threads_started":1}
WARNING: no policy specified for oryx/127.22.37.2@KRBTEST.COM; defaulting to no policy
I20260620 18:36:19.962643 29003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:19.941769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:46099 (local address 127.22.37.1:47877)
0620 18:36:19.942633 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:19.942688 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:19.945422 (+ 2734us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:19.946169 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:19.946184 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:19.946965 (+ 781us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:19.947903 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:19.947931 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:19.948948 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:19.948957 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:19.950468 (+ 1511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:19.950480 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:19.951575 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:19.951579 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:19.951769 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:19.952458 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:19.952485 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:19.954818 (+ 2333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:36:19.958781 (+ 3963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:19.958788 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:19.958801 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:19.959205 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:19.959635 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:19.959638 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:19.959640 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:19.959803 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:19.960484 (+ 681us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:36:19.960489 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:36:19.960782 (+ 293us) client_negotiation.cc:769] Sending connection context
0620 18:36:19.961003 (+ 221us) client_negotiation.cc:240] Negotiation successful
0620 18:36:19.961460 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":349,"threads_started":1}
I20260620 18:36:19.964335 29000 heartbeater.cc:347] Connected to a master server at 127.22.37.62:46099
I20260620 18:36:19.964689 29000 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.37.2@KRBTEST.COM" created.
I20260620 18:36:19.965550 29000 heartbeater.cc:511] Master 127.22.37.62:46099 requested a full tablet report, sending...
I20260620 18:36:19.967731 28811 ts_manager.cc:194] Registered new tserver with Master: 7232cd23c889418689f2c886ba008313 (127.22.37.1:35961)
I20260620 18:36:19.969161 28811 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.37.1@KRBTEST.COM'} at 127.22.37.1:47877
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.37.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/krb5kdc/oryx_127.22.37.2.keytab.
Entry for principal oryx/127.22.37.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/krb5kdc/oryx_127.22.37.2.keytab.
I20260620 18:36:19.988721 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.37.2:0
--local_ip_for_outbound_sockets=127.22.37.2
--webserver_interface=127.22.37.2
--webserver_port=0
--tserver_master_addrs=127.22.37.62:46099
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/krb5kdc/oryx_127.22.37.2.keytab
--principal=oryx/127.22.37.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:38175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:36:20.305774 29006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:36:20.306038 29006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:36:20.306227 29006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:36:20.312932 29006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:36:20.313021 29006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:36:20.313206 29006 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.37.2
I20260620 18:36:20.322402 29006 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:38175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/krb5kdc/oryx_127.22.37.2.keytab
--principal=oryx/127.22.37.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.37.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.37.2
--webserver_port=0
--tserver_master_addrs=127.22.37.62:46099
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.37.2
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:36:20.323596 29006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:36:20.324934 29006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:36:20.337714 29012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:20.337852 29011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:20.337800 29014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:20.338654 29006 server_base.cc:1034] running on GCE node
I20260620 18:36:20.339195 29006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:36:20.340154 29006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:36:20.341404 29006 hybrid_clock.cc:648] HybridClock initialized: now 1781980580341311 us; error 43 us; skew 500 ppm
I20260620 18:36:20.341732 29006 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:36:20 dist-test-slave-5lgp krb5kdc[28766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980580, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.37.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:36:20.344717 29006 init.cc:377] Logged in from keytab as oryx/127.22.37.2@KRBTEST.COM (short username oryx)
I20260620 18:36:20.346450 29006 webserver.cc:460] Webserver started at http://127.22.37.2:40713/ using document root <none> and password file <none>
I20260620 18:36:20.347292 29006 fs_manager.cc:359] Metadata directory not provided
I20260620 18:36:20.347436 29006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:36:20.347776 29006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:36:20.351545 29006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/ts-1/data/instance:
uuid: "519801f0627c402c9a65ccab106caf51"
format_stamp: "Formatted at 2026-06-20 18:36:20 on dist-test-slave-5lgp"
server_key: "184f67a051d793b3a6d90d320c0a173d"
server_key_iv: "f0ad1ae698cc5b84a59dd7be5030c73f"
server_key_version: "encryptionkey@0"
I20260620 18:36:20.352509 29006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/ts-1/wal/instance:
uuid: "519801f0627c402c9a65ccab106caf51"
format_stamp: "Formatted at 2026-06-20 18:36:20 on dist-test-slave-5lgp"
server_key: "184f67a051d793b3a6d90d320c0a173d"
server_key_iv: "f0ad1ae698cc5b84a59dd7be5030c73f"
server_key_version: "encryptionkey@0"
I20260620 18:36:20.358788 29006 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260620 18:36:20.362982 29021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:20.364480 29006 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 18:36:20.364619 29006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/ts-1/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/ts-1/wal
uuid: "519801f0627c402c9a65ccab106caf51"
format_stamp: "Formatted at 2026-06-20 18:36:20 on dist-test-slave-5lgp"
server_key: "184f67a051d793b3a6d90d320c0a173d"
server_key_iv: "f0ad1ae698cc5b84a59dd7be5030c73f"
server_key_version: "encryptionkey@0"
I20260620 18:36:20.364809 29006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:36:20.381083 29006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:36:20.385298 29006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:36:20.385646 29006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:36:20.386665 29006 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:36:20.388006 29006 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:36:20.388079 29006 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:20.388252 29006 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:36:20.388319 29006 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:20.422727 29006 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.2:33791
I20260620 18:36:20.422762 29133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.2:33791 every 8 connection(s)
I20260620 18:36:20.423991 29006 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/ts-1/data/info.pb
I20260620 18:36:20.428337 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 29006
I20260620 18:36:20.428552 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/ts-1/wal/instance
I20260620 18:36:20.429119 22676 external_mini_cluster.cc:1428] Setting key 32654d8a7bfdb9998cf3271826203d17
Jun 20 18:36:20 dist-test-slave-5lgp krb5kdc[28766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980580, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.37.2@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM
I20260620 18:36:20.445351 29002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:20.426686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.2:46559 (local address 127.22.37.62:46099)
0620 18:36:20.426893 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:20.426899 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:20.428124 (+ 1225us) server_negotiation.cc:407] Connection header received
0620 18:36:20.430423 (+ 2299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:20.430428 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:20.430502 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:20.430613 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:20.432532 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:20.433241 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:20.434775 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:20.435181 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:20.435315 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:20.438857 (+ 3542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:20.438862 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:20.438866 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:20.438896 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:20.442144 (+ 3248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:20.442852 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:20.442860 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:20.442866 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:20.443024 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:20.443428 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:20.443436 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:20.443442 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:20.443830 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:20.443981 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:20.444630 (+ 649us) server_negotiation.cc:299] Negotiation successful
0620 18:36:20.444892 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 18:36:20.446194 29136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:20.427219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:46099 (local address 127.22.37.2:46559)
0620 18:36:20.427923 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:20.427983 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:20.430145 (+ 2162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:20.430814 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:20.430823 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:20.431611 (+ 788us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:20.432361 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:20.432379 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:20.433413 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:20.433419 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:20.434625 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:20.434636 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:20.435401 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:20.435409 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:20.435643 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:20.436306 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:20.436332 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:20.438650 (+ 2318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:36:20.442288 (+ 3638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:20.442294 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:20.442308 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:20.442717 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:20.443141 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:20.443144 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:20.443146 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:20.443307 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:20.443945 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:36:20.443952 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:36:20.444258 (+ 306us) client_negotiation.cc:769] Sending connection context
0620 18:36:20.444488 (+ 230us) client_negotiation.cc:240] Negotiation successful
0620 18:36:20.444870 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":271,"threads_started":1}
I20260620 18:36:20.447682 29134 heartbeater.cc:347] Connected to a master server at 127.22.37.62:46099
I20260620 18:36:20.448040 29134 heartbeater.cc:464] Registering TS with master...
I20260620 18:36:20.449023 29134 heartbeater.cc:511] Master 127.22.37.62:46099 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.22.37.3@KRBTEST.COM; defaulting to no policy
I20260620 18:36:20.450745 28811 ts_manager.cc:194] Registered new tserver with Master: 519801f0627c402c9a65ccab106caf51 (127.22.37.2:33791)
I20260620 18:36:20.451735 28811 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.37.2@KRBTEST.COM'} at 127.22.37.2:46559
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.37.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.37.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/krb5kdc/oryx_127.22.37.3.keytab.
Entry for principal oryx/127.22.37.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/krb5kdc/oryx_127.22.37.3.keytab.
I20260620 18:36:20.476428 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.37.3:0
--local_ip_for_outbound_sockets=127.22.37.3
--webserver_interface=127.22.37.3
--webserver_port=0
--tserver_master_addrs=127.22.37.62:46099
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/krb5kdc/oryx_127.22.37.3.keytab
--principal=oryx/127.22.37.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:38175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:36:20.794152 29139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:36:20.794502 29139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:36:20.794667 29139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:36:20.802044 29139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:36:20.802183 29139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:36:20.802346 29139 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.37.3
I20260620 18:36:20.811533 29139 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:38175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/krb5kdc/oryx_127.22.37.3.keytab
--principal=oryx/127.22.37.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.37.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.37.3
--webserver_port=0
--tserver_master_addrs=127.22.37.62:46099
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.37.3
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:36:20.812834 29139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:36:20.814196 29139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:36:20.826923 29145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:20.826996 29147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:20.827028 29144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:20.827883 29139 server_base.cc:1034] running on GCE node
I20260620 18:36:20.828477 29139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:36:20.829487 29139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:36:20.830711 29139 hybrid_clock.cc:648] HybridClock initialized: now 1781980580830659 us; error 46 us; skew 500 ppm
I20260620 18:36:20.831033 29139 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:36:20 dist-test-slave-5lgp krb5kdc[28766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980580, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.37.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:36:20.834177 29139 init.cc:377] Logged in from keytab as oryx/127.22.37.3@KRBTEST.COM (short username oryx)
I20260620 18:36:20.835654 29139 webserver.cc:460] Webserver started at http://127.22.37.3:44253/ using document root <none> and password file <none>
I20260620 18:36:20.836299 29139 fs_manager.cc:359] Metadata directory not provided
I20260620 18:36:20.836405 29139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:36:20.836652 29139 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:36:20.839583 29139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/ts-2/data/instance:
uuid: "75525021a76043b8a1227ca817b38ec8"
format_stamp: "Formatted at 2026-06-20 18:36:20 on dist-test-slave-5lgp"
server_key: "71069479db998d2d45290018ed95bb3c"
server_key_iv: "14c7e1bfe4ce9098c85b0f762fd6f487"
server_key_version: "encryptionkey@0"
I20260620 18:36:20.840314 29139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/ts-2/wal/instance:
uuid: "75525021a76043b8a1227ca817b38ec8"
format_stamp: "Formatted at 2026-06-20 18:36:20 on dist-test-slave-5lgp"
server_key: "71069479db998d2d45290018ed95bb3c"
server_key_iv: "14c7e1bfe4ce9098c85b0f762fd6f487"
server_key_version: "encryptionkey@0"
I20260620 18:36:20.845695 29139 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 18:36:20.849155 29154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:20.850589 29139 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:36:20.850772 29139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/ts-2/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/ts-2/wal
uuid: "75525021a76043b8a1227ca817b38ec8"
format_stamp: "Formatted at 2026-06-20 18:36:20 on dist-test-slave-5lgp"
server_key: "71069479db998d2d45290018ed95bb3c"
server_key_iv: "14c7e1bfe4ce9098c85b0f762fd6f487"
server_key_version: "encryptionkey@0"
I20260620 18:36:20.850986 29139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:36:20.881620 29139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:36:20.886019 29139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:36:20.886320 29139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:36:20.887269 29139 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:36:20.888586 29139 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:36:20.888670 29139 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:20.888824 29139 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:36:20.888926 29139 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:20.920516 29139 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.3:35311
I20260620 18:36:20.920563 29266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.3:35311 every 8 connection(s)
I20260620 18:36:20.921813 29139 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/ts-2/data/info.pb
I20260620 18:36:20.925624 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 29139
I20260620 18:36:20.925789 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/ts-2/wal/instance
I20260620 18:36:20.926290 22676 external_mini_cluster.cc:1428] Setting key 5b2cbe53f1b3a7076f032a32c7bf9116
Jun 20 18:36:20 dist-test-slave-5lgp krb5kdc[28766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980580, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.37.3@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM
I20260620 18:36:20.942457 29002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:20.924376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.3:53311 (local address 127.22.37.62:46099)
0620 18:36:20.924559 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:20.924563 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:20.925844 (+ 1281us) server_negotiation.cc:407] Connection header received
0620 18:36:20.928083 (+ 2239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:20.928087 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:20.928188 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:20.928303 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:20.930096 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:20.930856 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:20.932329 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:20.932728 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:20.932822 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:20.936060 (+ 3238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:20.936066 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:20.936071 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:20.936109 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:20.939318 (+ 3209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:20.940028 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:20.940034 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:20.940037 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:20.940180 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:20.940684 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:20.940689 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:20.940692 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:20.941070 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:20.941215 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:20.942050 (+ 835us) server_negotiation.cc:299] Negotiation successful
0620 18:36:20.942209 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 18:36:20.943602 29269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:20.924798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:46099 (local address 127.22.37.3:53311)
0620 18:36:20.925587 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:20.925638 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:20.927806 (+ 2168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:20.928588 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:20.928599 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:20.929294 (+ 695us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:20.929955 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:20.929972 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:20.930984 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:20.930987 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:20.932214 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:20.932225 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:20.932920 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:20.932924 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:20.933134 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:20.933796 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:20.933817 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:20.935865 (+ 2048us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:36:20.939479 (+ 3614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:20.939486 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:20.939497 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:20.939883 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:20.940314 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:20.940320 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:20.940324 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:20.940579 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:20.941208 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:36:20.941215 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:36:20.941672 (+ 457us) client_negotiation.cc:769] Sending connection context
0620 18:36:20.941966 (+ 294us) client_negotiation.cc:240] Negotiation successful
0620 18:36:20.942497 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":317,"threads_started":1}
I20260620 18:36:20.944868 29267 heartbeater.cc:347] Connected to a master server at 127.22.37.62:46099
I20260620 18:36:20.945158 29267 heartbeater.cc:464] Registering TS with master...
I20260620 18:36:20.945919 29267 heartbeater.cc:511] Master 127.22.37.62:46099 requested a full tablet report, sending...
I20260620 18:36:20.947335 28811 ts_manager.cc:194] Registered new tserver with Master: 75525021a76043b8a1227ca817b38ec8 (127.22.37.3:35311)
I20260620 18:36:20.948125 28811 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.37.3@KRBTEST.COM'} at 127.22.37.3:53311
I20260620 18:36:20.953881 22676 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:36:20.972157 29000 heartbeater.cc:503] Master 127.22.37.62:46099 was elected leader, sending a full tablet report...
WARNING: no policy specified for oryx/127.22.37.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.37.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.37.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/krb5kdc/oryx_127.22.37.61.keytab.
Entry for principal oryx/127.22.37.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/krb5kdc/oryx_127.22.37.61.keytab.
I20260620 18:36:20.997205 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.61:44853
--webserver_interface=127.22.37.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/krb5kdc/oryx_127.22.37.61.keytab
--principal=oryx/127.22.37.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:38175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.37.62:46099,127.22.37.61:44853
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:36:21.320122 29272 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:36:21.320552 29272 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:36:21.320730 29272 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:36:21.328540 29272 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:36:21.328639 29272 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:36:21.328701 29272 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:36:21.328758 29272 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:36:21.328804 29272 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:36:21.339116 29272 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:38175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.22.37.62:46099,127.22.37.61:44853
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/krb5kdc/oryx_127.22.37.61.keytab
--principal=oryx/127.22.37.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.61:44853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.22.37.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
W20260620 18:36:21.339430 29272 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.22.37.62:46099,127.22.37.61:44853'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260620 18:36:21.340457 29272 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:36:21.341799 29272 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:36:21.354117 29278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:21.354211 29277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:21.354918 29280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:21.356773 29272 server_base.cc:1034] running on GCE node
I20260620 18:36:21.357669 29272 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:36:21.359040 29272 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:36:21.360296 29272 hybrid_clock.cc:648] HybridClock initialized: now 1781980581360235 us; error 50 us; skew 500 ppm
I20260620 18:36:21.360630 29272 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:36:21 dist-test-slave-5lgp krb5kdc[28766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980581, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.37.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:36:21.363629 29272 init.cc:377] Logged in from keytab as oryx/127.22.37.61@KRBTEST.COM (short username oryx)
I20260620 18:36:21.365173 29272 webserver.cc:460] Webserver started at http://127.22.37.61:38235/ using document root <none> and password file <none>
I20260620 18:36:21.365954 29272 fs_manager.cc:359] Metadata directory not provided
I20260620 18:36:21.366086 29272 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:36:21.366400 29272 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:36:21.369560 29272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/master-1/data/instance:
uuid: "b1f845ed42fb4dcd90063c45dedac043"
format_stamp: "Formatted at 2026-06-20 18:36:21 on dist-test-slave-5lgp"
server_key: "7918ea4882c6f8df4ff621e8b5023c99"
server_key_iv: "01ca865ccf83be9d83bb5643987f26de"
server_key_version: "encryptionkey@0"
I20260620 18:36:21.370378 29272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/master-1/wal/instance:
uuid: "b1f845ed42fb4dcd90063c45dedac043"
format_stamp: "Formatted at 2026-06-20 18:36:21 on dist-test-slave-5lgp"
server_key: "7918ea4882c6f8df4ff621e8b5023c99"
server_key_iv: "01ca865ccf83be9d83bb5643987f26de"
server_key_version: "encryptionkey@0"
I20260620 18:36:21.375603 29272 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:36:21.379177 29287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:21.380548 29272 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 18:36:21.380738 29272 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/master-1/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/master-1/wal
uuid: "b1f845ed42fb4dcd90063c45dedac043"
format_stamp: "Formatted at 2026-06-20 18:36:21 on dist-test-slave-5lgp"
server_key: "7918ea4882c6f8df4ff621e8b5023c99"
server_key_iv: "01ca865ccf83be9d83bb5643987f26de"
server_key_version: "encryptionkey@0"
I20260620 18:36:21.380960 29272 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:36:21.414790 29272 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:36:21.418628 29272 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:36:21.418885 29272 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:36:21.438825 29272 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.61:44853
I20260620 18:36:21.438843 29338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.61:44853 every 8 connection(s)
I20260620 18:36:21.440254 29272 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/master-1/data/info.pb
I20260620 18:36:21.443995 29339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:36:21.446780 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 29272
I20260620 18:36:21.451627 29339 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.37.62" port: 46099 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:36:21.454679 29134 heartbeater.cc:503] Master 127.22.37.62:46099 was elected leader, sending a full tablet report...
Jun 20 18:36:21 dist-test-slave-5lgp krb5kdc[28766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980581, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.37.61@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM
Jun 20 18:36:21 dist-test-slave-5lgp krb5kdc[28766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980578, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.37.61@KRBTEST.COM
I20260620 18:36:21.469784 29344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:21.454267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43814 (local address 127.22.37.62:46099)
0620 18:36:21.455566 (+ 1299us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:21.455574 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:21.455593 (+ 19us) server_negotiation.cc:407] Connection header received
0620 18:36:21.455654 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:21.455659 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:21.455735 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:21.455844 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:21.457140 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:21.457868 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:21.459496 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:21.459842 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:21.459968 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:21.463168 (+ 3200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:21.463172 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:21.463176 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:21.463204 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:21.466454 (+ 3250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:21.467249 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:21.467253 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:21.467256 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:21.467369 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:21.467848 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:21.467853 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:21.467856 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:21.468220 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:21.468362 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:21.469150 (+ 788us) server_negotiation.cc:299] Negotiation successful
0620 18:36:21.469408 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1177,"thread_start_us":1039,"threads_started":1}
I20260620 18:36:21.470790 29342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:21.448985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41112 (local address 127.22.37.61:44853)
0620 18:36:21.449749 (+ 764us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:21.449760 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:21.449802 (+ 42us) server_negotiation.cc:407] Connection header received
0620 18:36:21.451672 (+ 1870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:21.451694 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:21.452067 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:21.452405 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:21.454157 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:21.456936 (+ 2779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:21.458336 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:21.459035 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:21.459152 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:21.463649 (+ 4497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:21.463660 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:21.463675 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:21.463714 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:21.467019 (+ 3305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:21.467731 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:21.467737 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:21.467739 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:21.467915 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:21.468363 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:21.468368 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:21.468372 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:21.468725 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:21.468967 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:21.469395 (+ 428us) server_negotiation.cc:299] Negotiation successful
0620 18:36:21.469696 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":104,"server-negotiator.queue_time_us":576,"thread_start_us":360,"threads_started":1}
I20260620 18:36:21.470837 29343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:21.454223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:46099 (local address 127.0.0.1:43814)
0620 18:36:21.454856 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:21.454895 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:21.455031 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:21.455942 (+ 911us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:21.455949 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:21.456698 (+ 749us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:21.456969 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:21.456978 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:21.458003 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:21.458009 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:21.459361 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:21.459373 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:21.460072 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:21.460080 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:21.460217 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:21.460769 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:21.460788 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:21.462994 (+ 2206us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:36:21.466589 (+ 3595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:21.466601 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:21.466609 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:21.467136 (+ 527us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:21.467501 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:21.467506 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:21.467509 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:21.467743 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:21.468347 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:36:21.468358 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:36:21.468700 (+ 342us) client_negotiation.cc:769] Sending connection context
0620 18:36:21.469022 (+ 322us) client_negotiation.cc:240] Negotiation successful
0620 18:36:21.469576 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"mutex_wait_us":59,"thread_start_us":278,"threads_started":1}
I20260620 18:36:21.473273 29339 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.37.61" port: 44853 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:36:21 dist-test-slave-5lgp krb5kdc[28766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980581, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.37.61@KRBTEST.COM for oryx/127.22.37.61@KRBTEST.COM
I20260620 18:36:21.485430 29343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:21.474088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.61:44853 (local address 127.0.0.1:41128)
0620 18:36:21.474318 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:21.474332 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:21.474449 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:21.474724 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:21.474728 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:21.475069 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:21.475179 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:21.475187 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:21.476098 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:21.476103 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:21.476918 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:21.476928 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:21.477641 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:21.477646 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:21.477891 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:21.478371 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:21.478388 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:21.480220 (+ 1832us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:36:21.482928 (+ 2708us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:21.482936 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:21.482941 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:21.483367 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:21.483709 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:21.483713 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:21.483716 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:21.483838 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:21.484436 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:36:21.484440 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:36:21.484578 (+ 138us) client_negotiation.cc:769] Sending connection context
0620 18:36:21.484712 (+ 134us) client_negotiation.cc:240] Negotiation successful
0620 18:36:21.485001 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":36}
I20260620 18:36:21.485435 29342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:21.474057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41128 (local address 127.22.37.61:44853)
0620 18:36:21.474223 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:21.474230 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:21.474415 (+ 185us) server_negotiation.cc:407] Connection header received
0620 18:36:21.474573 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:21.474576 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:21.474622 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:21.474710 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:21.475303 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:21.475965 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:21.477051 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:21.477509 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:21.477635 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:21.480381 (+ 2746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:21.480385 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:21.480388 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:21.480414 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:21.482798 (+ 2384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:21.483475 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:21.483490 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:21.483494 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:21.483600 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:21.483938 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:21.483945 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:21.483948 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:21.484326 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:21.484458 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:21.484789 (+ 331us) server_negotiation.cc:299] Negotiation successful
0620 18:36:21.485118 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":171,"server-negotiator.queue_time_us":56}
I20260620 18:36:21.490367 29339 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1f845ed42fb4dcd90063c45dedac043: Bootstrap starting.
I20260620 18:36:21.493361 29339 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b1f845ed42fb4dcd90063c45dedac043: Neither blocks nor log segments found. Creating new log.
I20260620 18:36:21.494422 29339 log.cc:824] T 00000000000000000000000000000000 P b1f845ed42fb4dcd90063c45dedac043: Log is configured to *not* fsync() on all Append() calls
I20260620 18:36:21.496903 29339 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1f845ed42fb4dcd90063c45dedac043: No bootstrap required, opened a new log
I20260620 18:36:21.503705 29339 raft_consensus.cc:348] T 00000000000000000000000000000000 P b1f845ed42fb4dcd90063c45dedac043 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8fe34680f3c4d388068736f074cb839" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 46099 } } peers { permanent_uuid: "b1f845ed42fb4dcd90063c45dedac043" member_type: VOTER last_known_addr { host: "127.22.37.61" port: 44853 } }
I20260620 18:36:21.504043 29339 raft_consensus.cc:374] T 00000000000000000000000000000000 P b1f845ed42fb4dcd90063c45dedac043 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:36:21.504153 29339 raft_consensus.cc:729] T 00000000000000000000000000000000 P b1f845ed42fb4dcd90063c45dedac043 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1f845ed42fb4dcd90063c45dedac043, State: Initialized, Role: FOLLOWER
I20260620 18:36:21.504738 29339 consensus_queue.cc:260] T 00000000000000000000000000000000 P b1f845ed42fb4dcd90063c45dedac043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8fe34680f3c4d388068736f074cb839" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 46099 } } peers { permanent_uuid: "b1f845ed42fb4dcd90063c45dedac043" member_type: VOTER last_known_addr { host: "127.22.37.61" port: 44853 } }
I20260620 18:36:21.505569 29346 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1f845ed42fb4dcd90063c45dedac043 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8fe34680f3c4d388068736f074cb839" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 46099 } } peers { permanent_uuid: "b1f845ed42fb4dcd90063c45dedac043" member_type: VOTER last_known_addr { host: "127.22.37.61" port: 44853 } } }
I20260620 18:36:21.505844 29346 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1f845ed42fb4dcd90063c45dedac043 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:36:21.506570 29339 sys_catalog.cc:564] T 00000000000000000000000000000000 P b1f845ed42fb4dcd90063c45dedac043 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:36:21.508956 29346 raft_consensus.cc:482] T 00000000000000000000000000000000 P b1f845ed42fb4dcd90063c45dedac043 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:36:21.509210 29346 raft_consensus.cc:504] T 00000000000000000000000000000000 P b1f845ed42fb4dcd90063c45dedac043 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8fe34680f3c4d388068736f074cb839" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 46099 } } peers { permanent_uuid: "b1f845ed42fb4dcd90063c45dedac043" member_type: VOTER last_known_addr { host: "127.22.37.61" port: 44853 } }
I20260620 18:36:21.510619 29346 leader_election.cc:288] T 00000000000000000000000000000000 P b1f845ed42fb4dcd90063c45dedac043 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8fe34680f3c4d388068736f074cb839 (127.22.37.62:46099)
I20260620 18:36:21.511489 28821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b1f845ed42fb4dcd90063c45dedac043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8fe34680f3c4d388068736f074cb839" is_pre_election: true
I20260620 18:36:21.512600 29288 leader_election.cc:302] T 00000000000000000000000000000000 P b1f845ed42fb4dcd90063c45dedac043 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: b1f845ed42fb4dcd90063c45dedac043; no voters: c8fe34680f3c4d388068736f074cb839
I20260620 18:36:21.513005 29346 raft_consensus.cc:2726] T 00000000000000000000000000000000 P b1f845ed42fb4dcd90063c45dedac043 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260620 18:36:21.515504 29357 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b1f845ed42fb4dcd90063c45dedac043: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 18:36:21.515630 29357 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 18:36:21.517556 29272 master_runner.cc:428] Detected that this master b1f845ed42fb4dcd90063c45dedac043 is joining an existing cluster
I20260620 18:36:21.517642 29272 master_runner.cc:432] Initiating AddMaster RPC to add 127.22.37.61:44853
I20260620 18:36:21.537642 29344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:21.521418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43818 (local address 127.22.37.62:46099)
0620 18:36:21.521668 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:21.521677 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:21.522601 (+ 924us) server_negotiation.cc:407] Connection header received
0620 18:36:21.522798 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:21.522805 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:21.522907 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:21.523008 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:21.524626 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:21.525776 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:21.527773 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:21.528409 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:21.528504 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:21.530976 (+ 2472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:21.530995 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:21.531000 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:21.531038 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:21.534465 (+ 3427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:21.535333 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:21.535339 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:21.535343 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:21.535486 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:21.535980 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:21.535985 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:21.535988 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:21.536333 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:21.536504 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:21.537024 (+ 520us) server_negotiation.cc:299] Negotiation successful
0620 18:36:21.537279 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 18:36:21.537770 29363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:21.521586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:46099 (local address 127.0.0.1:43818)
0620 18:36:21.522487 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:21.522507 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:21.522663 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:21.523185 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:21.523190 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:21.523986 (+ 796us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:21.524479 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:21.524490 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:21.525942 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:21.525947 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:21.527604 (+ 1657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:21.527619 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:21.528784 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:21.528790 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:21.528947 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:21.529637 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:21.529658 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:21.530760 (+ 1102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:36:21.534609 (+ 3849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:21.534614 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:21.534618 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:21.535194 (+ 576us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:21.535642 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:21.535647 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:21.535652 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:21.535822 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:21.536464 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:36:21.536470 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:36:21.536665 (+ 195us) client_negotiation.cc:769] Sending connection context
0620 18:36:21.536911 (+ 246us) client_negotiation.cc:240] Negotiation successful
0620 18:36:21.537215 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":772,"mutex_wait_us":4,"spinlock_wait_cycles":25472,"thread_start_us":626,"threads_started":1}
I20260620 18:36:21.538641 29342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:21.521791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41144 (local address 127.22.37.61:44853)
0620 18:36:21.522028 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:21.522036 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:21.523303 (+ 1267us) server_negotiation.cc:407] Connection header received
0620 18:36:21.523449 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:21.523455 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:21.523532 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:21.523636 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:21.525328 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:21.526487 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:21.528872 (+ 2385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:21.529638 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:21.529738 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:21.532244 (+ 2506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:21.532252 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:21.532260 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:21.532295 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:21.535752 (+ 3457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:21.536520 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:21.536526 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:21.536529 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:21.536698 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:21.537089 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:21.537093 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:21.537096 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:21.537506 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:21.537616 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:21.538106 (+ 490us) server_negotiation.cc:299] Negotiation successful
0620 18:36:21.538311 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75,"spinlock_wait_cycles":17152}
I20260620 18:36:21.539474 29364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:21.521649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.61:44853 (local address 127.0.0.1:41144)
0620 18:36:21.523200 (+ 1551us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:21.523216 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:21.523332 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:21.523677 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:21.523683 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:21.524848 (+ 1165us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:21.525182 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:21.525195 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:21.526722 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:21.526730 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:21.528717 (+ 1987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:21.528735 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:21.530042 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:21.530050 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:21.530261 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:21.530856 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:21.530881 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:21.532081 (+ 1200us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:36:21.535885 (+ 3804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:21.535892 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:21.535898 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:21.536403 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:21.536800 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:21.536802 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:21.536804 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:21.536972 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:21.537699 (+ 727us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:36:21.537706 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:36:21.537874 (+ 168us) client_negotiation.cc:769] Sending connection context
0620 18:36:21.538859 (+ 985us) client_negotiation.cc:240] Negotiation successful
0620 18:36:21.539127 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1430,"mutex_wait_us":1,"thread_start_us":1239,"threads_started":1}
Jun 20 18:36:21 dist-test-slave-5lgp krb5kdc[28766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980579, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.37.62@KRBTEST.COM for oryx/127.22.37.61@KRBTEST.COM
WARNING: no policy specified for oryx/127.22.37.60@KRBTEST.COM; defaulting to no policy
I20260620 18:36:21.558938 29366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:21.544055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.61:44853 (local address 127.0.0.1:41160)
0620 18:36:21.545870 (+ 1815us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:21.545911 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:21.546108 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:21.546575 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:21.546582 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:21.547313 (+ 731us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:21.547615 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:21.547626 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:21.548682 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:21.548686 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:21.549761 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:21.549772 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:21.550536 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:21.550542 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:21.550721 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:21.551274 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:21.551290 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:21.553459 (+ 2169us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:36:21.556590 (+ 3131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:21.556601 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:21.556608 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:21.556978 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:21.557349 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:21.557353 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:21.557357 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:21.557519 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:21.558022 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:36:21.558030 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:36:21.558198 (+ 168us) client_negotiation.cc:769] Sending connection context
0620 18:36:21.558395 (+ 197us) client_negotiation.cc:240] Negotiation successful
0620 18:36:21.558636 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1615,"thread_start_us":1476,"threads_started":1}
I20260620 18:36:21.559129 29342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:21.544055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41160 (local address 127.22.37.61:44853)
0620 18:36:21.544224 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:21.544232 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:21.545986 (+ 1754us) server_negotiation.cc:407] Connection header received
0620 18:36:21.546324 (+ 338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:21.546329 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:21.546419 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:21.546528 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:21.547788 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:21.548540 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:21.549898 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:21.550380 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:21.550518 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:21.553630 (+ 3112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:21.553636 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:21.553639 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:21.553666 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:21.556453 (+ 2787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:21.557100 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:21.557105 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:21.557109 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:21.557234 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:21.557643 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:21.557648 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:21.557651 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:21.557899 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:21.558043 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:21.558530 (+ 487us) server_negotiation.cc:299] Negotiation successful
0620 18:36:21.558750 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.37.60@KRBTEST.COM" created.
I20260620 18:36:21.561013 28811 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.22.37.61:44853: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "b1f845ed42fb4dcd90063c45dedac043"
member_type: NON_VOTER
last_known_addr {
host: "127.22.37.61"
port: 44853
}
attrs {
promote: true
}
}
dest_uuid: "c8fe34680f3c4d388068736f074cb839"
cas_config_opid_index: -1
I20260620 18:36:21.562040 28811 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8fe34680f3c4d388068736f074cb839 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c8fe34680f3c4d388068736f074cb839" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 46099 } } peers { permanent_uuid: "b1f845ed42fb4dcd90063c45dedac043" member_type: NON_VOTER last_known_addr { host: "127.22.37.61" port: 44853 } attrs { promote: true } }
I20260620 18:36:21.578342 29314 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b1f845ed42fb4dcd90063c45dedac043 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:36:21.583913 29314 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b1f845ed42fb4dcd90063c45dedac043 [term 1 FOLLOWER]: Refusing update from remote peer c8fe34680f3c4d388068736f074cb839: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260620 18:36:21.586001 29369 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c8fe34680f3c4d388068736f074cb839 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1f845ed42fb4dcd90063c45dedac043" member_type: NON_VOTER last_known_addr { host: "127.22.37.61" port: 44853 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:36:21.585634 29371 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c8fe34680f3c4d388068736f074cb839 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER b1f845ed42fb4dcd90063c45dedac043 (127.22.37.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c8fe34680f3c4d388068736f074cb839" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 46099 } } peers { permanent_uuid: "b1f845ed42fb4dcd90063c45dedac043" member_type: NON_VOTER last_known_addr { host: "127.22.37.61" port: 44853 } attrs { promote: true } } }
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.37.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/krb5kdc/oryx_127.22.37.60.keytab.
Entry for principal oryx/127.22.37.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/krb5kdc/oryx_127.22.37.60.keytab.
I20260620 18:36:21.591511 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.60:44179
--webserver_interface=127.22.37.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/krb5kdc/oryx_127.22.37.60.keytab
--principal=oryx/127.22.37.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:38175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.37.62:46099,127.22.37.61:44853,127.22.37.60:44179
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:36:21.598289 29371 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.22.37.61:44853
I20260620 18:36:21.599093 29272 master.cc:537] Master@127.22.37.61:44853 shutting down...
I20260620 18:36:21.599772 29369 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8fe34680f3c4d388068736f074cb839 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c8fe34680f3c4d388068736f074cb839" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c8fe34680f3c4d388068736f074cb839" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 46099 } } peers { permanent_uuid: "b1f845ed42fb4dcd90063c45dedac043" member_type: NON_VOTER last_known_addr { host: "127.22.37.61" port: 44853 } attrs { promote: true } } }
I20260620 18:36:21.600070 29369 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8fe34680f3c4d388068736f074cb839 [sys.catalog]: This master's current role is: LEADER
I20260620 18:36:21.616916 29314 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b1f845ed42fb4dcd90063c45dedac043 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER b1f845ed42fb4dcd90063c45dedac043 (127.22.37.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c8fe34680f3c4d388068736f074cb839" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 46099 } } peers { permanent_uuid: "b1f845ed42fb4dcd90063c45dedac043" member_type: NON_VOTER last_known_addr { host: "127.22.37.61" port: 44853 } attrs { promote: true } } }
I20260620 18:36:21.633625 29346 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1f845ed42fb4dcd90063c45dedac043 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8fe34680f3c4d388068736f074cb839. Latest consensus state: current_term: 1 leader_uuid: "c8fe34680f3c4d388068736f074cb839" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c8fe34680f3c4d388068736f074cb839" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 46099 } } peers { permanent_uuid: "b1f845ed42fb4dcd90063c45dedac043" member_type: NON_VOTER last_known_addr { host: "127.22.37.61" port: 44853 } attrs { promote: true } } }
I20260620 18:36:21.633881 29346 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1f845ed42fb4dcd90063c45dedac043 [sys.catalog]: This master's current role is: LEARNER
I20260620 18:36:21.634004 29380 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1f845ed42fb4dcd90063c45dedac043 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c8fe34680f3c4d388068736f074cb839" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c8fe34680f3c4d388068736f074cb839" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 46099 } } peers { permanent_uuid: "b1f845ed42fb4dcd90063c45dedac043" member_type: NON_VOTER last_known_addr { host: "127.22.37.61" port: 44853 } attrs { promote: true } } }
I20260620 18:36:21.634667 29380 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1f845ed42fb4dcd90063c45dedac043 [sys.catalog]: This master's current role is: LEARNER
I20260620 18:36:21.634258 29374 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1f845ed42fb4dcd90063c45dedac043 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c8fe34680f3c4d388068736f074cb839" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c8fe34680f3c4d388068736f074cb839" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 46099 } } peers { permanent_uuid: "b1f845ed42fb4dcd90063c45dedac043" member_type: NON_VOTER last_known_addr { host: "127.22.37.61" port: 44853 } attrs { promote: true } } }
I20260620 18:36:21.635648 29374 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1f845ed42fb4dcd90063c45dedac043 [sys.catalog]: This master's current role is: LEARNER
I20260620 18:36:21.636938 29272 raft_consensus.cc:2229] T 00000000000000000000000000000000 P b1f845ed42fb4dcd90063c45dedac043 [term 1 LEARNER]: Raft consensus shutting down.
I20260620 18:36:21.637359 29272 raft_consensus.cc:2258] T 00000000000000000000000000000000 P b1f845ed42fb4dcd90063c45dedac043 [term 1 LEARNER]: Raft consensus is shut down!
I20260620 18:36:21.637574 29272 tablet_replica.cc:330] T 00000000000000000000000000000000 P b1f845ed42fb4dcd90063c45dedac043: stopping tablet replica
I20260620 18:36:21.654665 29272 master.cc:552] Master@127.22.37.61:44853 shutdown complete.
I20260620 18:36:21.654781 29272 master_runner.cc:305] Clearing existing system tablet
I20260620 18:36:21.655221 29272 fs_manager.cc:359] Metadata directory not provided
I20260620 18:36:21.655339 29272 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:36:21.657763 29272 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260620 18:36:21.659229 29382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:21.659590 29272 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:36:21.659890 29272 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:36:21.660004 29272 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/master-1/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/master-1/wal
uuid: "b1f845ed42fb4dcd90063c45dedac043"
format_stamp: "Formatted at 2026-06-20 18:36:21 on dist-test-slave-5lgp"
server_key: "7918ea4882c6f8df4ff621e8b5023c99"
server_key_iv: "01ca865ccf83be9d83bb5643987f26de"
server_key_version: "encryptionkey@0"
I20260620 18:36:21.663151 29272 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P b1f845ed42fb4dcd90063c45dedac043: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:36:21.667861 29272 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P b1f845ed42fb4dcd90063c45dedac043: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 18:36:21.668095 29272 log.cc:1196] T 00000000000000000000000000000000 P b1f845ed42fb4dcd90063c45dedac043: Deleting WAL directory at /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260620 18:36:21.668663 29272 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P b1f845ed42fb4dcd90063c45dedac043: Deleting consensus metadata
I20260620 18:36:21.668915 29272 master_runner.cc:315] Copying system tablet from 127.22.37.62:46099
I20260620 18:36:21.673826 29272 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P b1f845ed42fb4dcd90063c45dedac043: tablet copy: Beginning tablet copy session from remote peer at address 127.22.37.62:46099
I20260620 18:36:21.686745 29344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:21.675089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43830 (local address 127.22.37.62:46099)
0620 18:36:21.675261 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:21.675270 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:21.675367 (+ 97us) server_negotiation.cc:407] Connection header received
0620 18:36:21.675667 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:21.675673 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:21.675749 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:21.675861 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:21.676679 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:21.677474 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:21.678710 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:21.679120 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:21.679245 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:21.680959 (+ 1714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:21.680964 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:21.680969 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:21.681010 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:21.684066 (+ 3056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:21.684652 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:21.684659 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:21.684662 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:21.684798 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:21.685164 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:21.685171 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:21.685175 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:21.685520 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:21.685671 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:21.686155 (+ 484us) server_negotiation.cc:299] Negotiation successful
0620 18:36:21.686355 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 18:36:21.686779 29388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:21.674629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:46099 (local address 127.0.0.1:43830)
0620 18:36:21.675257 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:21.675281 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:21.675482 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:21.675888 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:21.675892 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:21.676316 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:21.676531 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:21.676540 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:21.677621 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:21.677626 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:21.678555 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:21.678565 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:21.679266 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:21.679272 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:21.679462 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:21.680089 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:21.680105 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:21.680776 (+ 671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:36:21.684180 (+ 3404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:21.684186 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:21.684191 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:21.684538 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:21.684907 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:21.684912 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:21.684919 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:21.685059 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:21.685634 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:36:21.685639 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:36:21.685867 (+ 228us) client_negotiation.cc:769] Sending connection context
0620 18:36:21.686060 (+ 193us) client_negotiation.cc:240] Negotiation successful
0620 18:36:21.686363 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":640,"threads_started":1}
I20260620 18:36:21.687844 28831 tablet_copy_service.cc:140] P c8fe34680f3c4d388068736f074cb839: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer b1f845ed42fb4dcd90063c45dedac043 ({username='oryx', principal='oryx/127.22.37.61@KRBTEST.COM'} at 127.0.0.1:43830)
I20260620 18:36:21.688062 28831 tablet_copy_service.cc:161] P c8fe34680f3c4d388068736f074cb839: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer b1f845ed42fb4dcd90063c45dedac043 at {username='oryx', principal='oryx/127.22.37.61@KRBTEST.COM'} at 127.0.0.1:43830: session id = b1f845ed42fb4dcd90063c45dedac043-00000000000000000000000000000000
I20260620 18:36:21.691509 28831 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P c8fe34680f3c4d388068736f074cb839: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 18:36:21.693477 29272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 18:36:21.698194 29272 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P b1f845ed42fb4dcd90063c45dedac043: tablet copy: Starting download of 0 data blocks...
I20260620 18:36:21.698484 29272 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P b1f845ed42fb4dcd90063c45dedac043: tablet copy: Starting download of 1 WAL segments...
I20260620 18:36:21.700781 29272 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P b1f845ed42fb4dcd90063c45dedac043: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 18:36:21.704195 28831 tablet_copy_service.cc:342] P c8fe34680f3c4d388068736f074cb839: Request end of tablet copy session b1f845ed42fb4dcd90063c45dedac043-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.22.37.61@KRBTEST.COM'} at 127.0.0.1:43830
I20260620 18:36:21.704381 28831 tablet_copy_service.cc:433] P c8fe34680f3c4d388068736f074cb839: ending tablet copy session b1f845ed42fb4dcd90063c45dedac043-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer b1f845ed42fb4dcd90063c45dedac043
I20260620 18:36:21.714790 29272 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:36:21.717480 29389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:21.723551 29390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:21.728328 29392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:21.730458 29272 server_base.cc:1034] running on GCE node
I20260620 18:36:21.730809 29272 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:36:21.731521 29272 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:36:21.732796 29272 hybrid_clock.cc:648] HybridClock initialized: now 1781980581732743 us; error 35 us; skew 500 ppm
I20260620 18:36:21.733031 29272 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:36:21 dist-test-slave-5lgp krb5kdc[28766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980581, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.37.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:36:21.735103 29272 init.cc:377] Logged in from keytab as oryx/127.22.37.61@KRBTEST.COM (short username oryx)
I20260620 18:36:21.741097 29272 webserver.cc:460] Webserver started at http://127.22.37.61:42781/ using document root <none> and password file <none>
I20260620 18:36:21.741549 29272 fs_manager.cc:359] Metadata directory not provided
I20260620 18:36:21.741654 29272 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:36:21.744122 29272 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20260620 18:36:21.745843 29399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:21.746232 29272 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 18:36:21.746385 29272 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/master-1/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/master-1/wal
uuid: "b1f845ed42fb4dcd90063c45dedac043"
format_stamp: "Formatted at 2026-06-20 18:36:21 on dist-test-slave-5lgp"
server_key: "7918ea4882c6f8df4ff621e8b5023c99"
server_key_iv: "01ca865ccf83be9d83bb5643987f26de"
server_key_version: "encryptionkey@0"
I20260620 18:36:21.746553 29272 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:36:21.772341 29272 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:36:21.776021 29272 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:36:21.801796 29272 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.61:44853
I20260620 18:36:21.801810 29450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.61:44853 every 8 connection(s)
I20260620 18:36:21.802750 29272 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/master-1/data/info.pb
I20260620 18:36:21.810058 29451 sys_catalog.cc:263] Verifying existing consensus state
I20260620 18:36:21.814534 29451 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1f845ed42fb4dcd90063c45dedac043: Bootstrap starting.
I20260620 18:36:21.824764 29451 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1f845ed42fb4dcd90063c45dedac043: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 18:36:21.825259 29451 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1f845ed42fb4dcd90063c45dedac043: Bootstrap complete.
I20260620 18:36:21.826198 29451 raft_consensus.cc:348] T 00000000000000000000000000000000 P b1f845ed42fb4dcd90063c45dedac043 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c8fe34680f3c4d388068736f074cb839" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 46099 } } peers { permanent_uuid: "b1f845ed42fb4dcd90063c45dedac043" member_type: NON_VOTER last_known_addr { host: "127.22.37.61" port: 44853 } attrs { promote: true } }
I20260620 18:36:21.826413 29451 raft_consensus.cc:729] T 00000000000000000000000000000000 P b1f845ed42fb4dcd90063c45dedac043 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b1f845ed42fb4dcd90063c45dedac043, State: Initialized, Role: LEARNER
I20260620 18:36:21.826673 29451 consensus_queue.cc:260] T 00000000000000000000000000000000 P b1f845ed42fb4dcd90063c45dedac043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c8fe34680f3c4d388068736f074cb839" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 46099 } } peers { permanent_uuid: "b1f845ed42fb4dcd90063c45dedac043" member_type: NON_VOTER last_known_addr { host: "127.22.37.61" port: 44853 } attrs { promote: true } }
I20260620 18:36:21.827575 29454 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1f845ed42fb4dcd90063c45dedac043 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c8fe34680f3c4d388068736f074cb839" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 46099 } } peers { permanent_uuid: "b1f845ed42fb4dcd90063c45dedac043" member_type: NON_VOTER last_known_addr { host: "127.22.37.61" port: 44853 } attrs { promote: true } } }
I20260620 18:36:21.827855 29451 sys_catalog.cc:564] T 00000000000000000000000000000000 P b1f845ed42fb4dcd90063c45dedac043 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:36:21.828168 29454 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1f845ed42fb4dcd90063c45dedac043 [sys.catalog]: This master's current role is: LEARNER
I20260620 18:36:21.837850 29465 catalog_manager.cc:1259] Loaded cluster ID: eee720b350864e718f512e0ff17d1a2e
I20260620 18:36:21.837970 29465 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b1f845ed42fb4dcd90063c45dedac043: loading cluster ID for follower catalog manager: success
I20260620 18:36:21.840283 29465 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b1f845ed42fb4dcd90063c45dedac043: acquiring CA information for follower catalog manager: success
I20260620 18:36:21.842178 29465 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b1f845ed42fb4dcd90063c45dedac043: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 18:36:21.950505 29267 heartbeater.cc:503] Master 127.22.37.62:46099 was elected leader, sending a full tablet report...
W20260620 18:36:22.027387 29375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:36:22.027743 29375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:36:22.027904 29375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:36:22.035701 29375 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:36:22.035846 29375 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:36:22.035959 29375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:36:22.036053 29375 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:36:22.036139 29375 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:36:22.045796 29375 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:38175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.22.37.62:46099,127.22.37.61:44853,127.22.37.60:44179
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/krb5kdc/oryx_127.22.37.60.keytab
--principal=oryx/127.22.37.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.60:44179
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.22.37.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:36:22.047173 29375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:36:22.048549 29375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 18:36:22.058102 29368 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8fe34680f3c4d388068736f074cb839 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c8fe34680f3c4d388068736f074cb839" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c8fe34680f3c4d388068736f074cb839" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 46099 } } peers { permanent_uuid: "b1f845ed42fb4dcd90063c45dedac043" member_type: NON_VOTER last_known_addr { host: "127.22.37.61" port: 44853 } attrs { promote: true } } }
I20260620 18:36:22.058521 29368 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8fe34680f3c4d388068736f074cb839 [sys.catalog]: This master's current role is: LEADER
W20260620 18:36:22.062655 29471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:22.062917 29473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:22.063342 29470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:22.064419 29375 server_base.cc:1034] running on GCE node
I20260620 18:36:22.065193 29375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:36:22.066447 29474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:22.058655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41170 (local address 127.22.37.61:44853)
0620 18:36:22.059277 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:22.059282 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:22.059298 (+ 16us) server_negotiation.cc:407] Connection header received
0620 18:36:22.059357 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:22.059360 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:22.059421 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:22.059569 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:36:22.060056 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:22.061850 (+ 1794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:22.064621 (+ 2771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:22.065532 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:22.065627 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:22.065864 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:22.066105 (+ 241us) server_negotiation.cc:299] Negotiation successful
0620 18:36:22.066194 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":507,"thread_start_us":317,"threads_started":1}
I20260620 18:36:22.066452 29366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:22.058411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.61:44853 (local address 127.0.0.1:41170)
0620 18:36:22.058607 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:22.058632 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:22.058772 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:22.059597 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:22.059602 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:22.059632 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:36:22.059863 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:22.059877 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:22.061992 (+ 2115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:22.061997 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:22.064448 (+ 2451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:22.064464 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:22.065690 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:22.065696 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:22.065934 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:22.065988 (+ 54us) client_negotiation.cc:769] Sending connection context
0620 18:36:22.066090 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 18:36:22.066214 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260620 18:36:22.066949 29375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:36:22.067593 29454 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1f845ed42fb4dcd90063c45dedac043 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8fe34680f3c4d388068736f074cb839. Latest consensus state: current_term: 1 leader_uuid: "c8fe34680f3c4d388068736f074cb839" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c8fe34680f3c4d388068736f074cb839" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 46099 } } peers { permanent_uuid: "b1f845ed42fb4dcd90063c45dedac043" member_type: NON_VOTER last_known_addr { host: "127.22.37.61" port: 44853 } attrs { promote: true } } }
I20260620 18:36:22.067888 29454 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1f845ed42fb4dcd90063c45dedac043 [sys.catalog]: This master's current role is: LEARNER
I20260620 18:36:22.068184 29375 hybrid_clock.cc:648] HybridClock initialized: now 1781980582068080 us; error 92 us; skew 500 ppm
I20260620 18:36:22.068436 29369 raft_consensus.cc:1050] T 00000000000000000000000000000000 P c8fe34680f3c4d388068736f074cb839: attempting to promote NON_VOTER b1f845ed42fb4dcd90063c45dedac043 to VOTER
I20260620 18:36:22.068650 29375 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:36:22.069442 29369 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8fe34680f3c4d388068736f074cb839 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c8fe34680f3c4d388068736f074cb839" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 46099 } } peers { permanent_uuid: "b1f845ed42fb4dcd90063c45dedac043" member_type: VOTER last_known_addr { host: "127.22.37.61" port: 44853 } attrs { promote: false } }
I20260620 18:36:22.071158 29426 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b1f845ed42fb4dcd90063c45dedac043 [term 1 LEARNER]: Refusing update from remote peer c8fe34680f3c4d388068736f074cb839: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 18:36:22.071735 29368 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c8fe34680f3c4d388068736f074cb839 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1f845ed42fb4dcd90063c45dedac043" member_type: VOTER last_known_addr { host: "127.22.37.61" port: 44853 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
Jun 20 18:36:22 dist-test-slave-5lgp krb5kdc[28766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980582, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.37.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:36:22.072577 29375 init.cc:377] Logged in from keytab as oryx/127.22.37.60@KRBTEST.COM (short username oryx)
I20260620 18:36:22.074775 29369 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c8fe34680f3c4d388068736f074cb839 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b1f845ed42fb4dcd90063c45dedac043 (127.22.37.61) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c8fe34680f3c4d388068736f074cb839" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 46099 } } peers { permanent_uuid: "b1f845ed42fb4dcd90063c45dedac043" member_type: VOTER last_known_addr { host: "127.22.37.61" port: 44853 } attrs { promote: false } } }
I20260620 18:36:22.075425 29426 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b1f845ed42fb4dcd90063c45dedac043 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b1f845ed42fb4dcd90063c45dedac043 (127.22.37.61) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c8fe34680f3c4d388068736f074cb839" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 46099 } } peers { permanent_uuid: "b1f845ed42fb4dcd90063c45dedac043" member_type: VOTER last_known_addr { host: "127.22.37.61" port: 44853 } attrs { promote: false } } }
I20260620 18:36:22.076313 29375 webserver.cc:460] Webserver started at http://127.22.37.60:33155/ using document root <none> and password file <none>
I20260620 18:36:22.077258 29375 fs_manager.cc:359] Metadata directory not provided
I20260620 18:36:22.077412 29375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:36:22.077749 29375 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:36:22.077775 29369 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8fe34680f3c4d388068736f074cb839 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c8fe34680f3c4d388068736f074cb839" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c8fe34680f3c4d388068736f074cb839" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 46099 } } peers { permanent_uuid: "b1f845ed42fb4dcd90063c45dedac043" member_type: VOTER last_known_addr { host: "127.22.37.61" port: 44853 } attrs { promote: false } } }
I20260620 18:36:22.077996 29369 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8fe34680f3c4d388068736f074cb839 [sys.catalog]: This master's current role is: LEADER
I20260620 18:36:22.078310 29371 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8fe34680f3c4d388068736f074cb839 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c8fe34680f3c4d388068736f074cb839" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c8fe34680f3c4d388068736f074cb839" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 46099 } } peers { permanent_uuid: "b1f845ed42fb4dcd90063c45dedac043" member_type: VOTER last_known_addr { host: "127.22.37.61" port: 44853 } attrs { promote: false } } }
I20260620 18:36:22.078475 29371 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8fe34680f3c4d388068736f074cb839 [sys.catalog]: This master's current role is: LEADER
I20260620 18:36:22.079675 29454 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1f845ed42fb4dcd90063c45dedac043 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c8fe34680f3c4d388068736f074cb839" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c8fe34680f3c4d388068736f074cb839" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 46099 } } peers { permanent_uuid: "b1f845ed42fb4dcd90063c45dedac043" member_type: VOTER last_known_addr { host: "127.22.37.61" port: 44853 } attrs { promote: false } } }
I20260620 18:36:22.079883 29454 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1f845ed42fb4dcd90063c45dedac043 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:36:22.081295 29375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/master-2/data/instance:
uuid: "e3b608bf35ad4fa2ab340274d7920a6c"
format_stamp: "Formatted at 2026-06-20 18:36:22 on dist-test-slave-5lgp"
server_key: "6beb82d6f924561048c1fbb60fe5770e"
server_key_iv: "cc8349e51006b0a33fb931cd3902f918"
server_key_version: "encryptionkey@0"
I20260620 18:36:22.082093 29375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/master-2/wal/instance:
uuid: "e3b608bf35ad4fa2ab340274d7920a6c"
format_stamp: "Formatted at 2026-06-20 18:36:22 on dist-test-slave-5lgp"
server_key: "6beb82d6f924561048c1fbb60fe5770e"
server_key_iv: "cc8349e51006b0a33fb931cd3902f918"
server_key_version: "encryptionkey@0"
I20260620 18:36:22.087306 29375 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:36:22.090659 29482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:22.091912 29375 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:36:22.092079 29375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/master-2/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/master-2/wal
uuid: "e3b608bf35ad4fa2ab340274d7920a6c"
format_stamp: "Formatted at 2026-06-20 18:36:22 on dist-test-slave-5lgp"
server_key: "6beb82d6f924561048c1fbb60fe5770e"
server_key_iv: "cc8349e51006b0a33fb931cd3902f918"
server_key_version: "encryptionkey@0"
I20260620 18:36:22.092260 29375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:36:22.112028 29375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:36:22.116294 29375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:36:22.116580 29375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:36:22.134940 29533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.60:44179 every 8 connection(s)
I20260620 18:36:22.134953 29375 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.60:44179
I20260620 18:36:22.136299 29375 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/master-2/data/info.pb
I20260620 18:36:22.139726 29534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:36:22.140237 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 29375
I20260620 18:36:22.148439 29534 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.37.62" port: 46099 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:36:22 dist-test-slave-5lgp krb5kdc[28766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980578, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.37.60@KRBTEST.COM
Jun 20 18:36:22 dist-test-slave-5lgp krb5kdc[28766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980582, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.37.60@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM
I20260620 18:36:22.162343 29537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:22.142239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40494 (local address 127.22.37.60:44179)
0620 18:36:22.142965 (+ 726us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:22.142975 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:22.143020 (+ 45us) server_negotiation.cc:407] Connection header received
0620 18:36:22.144411 (+ 1391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:22.144430 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:22.144784 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:22.145163 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:22.146645 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:22.148134 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:22.149590 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:22.150219 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:22.150402 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:22.155316 (+ 4914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:22.155324 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:22.155334 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:22.155365 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:22.158437 (+ 3072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:22.159207 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:22.159215 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:22.159223 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:22.159410 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:22.159809 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:22.159814 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:22.159817 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:22.160264 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:22.160553 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:22.160979 (+ 426us) server_negotiation.cc:299] Negotiation successful
0620 18:36:22.161341 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":318,"threads_started":1}
I20260620 18:36:22.165654 29344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:22.150932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43840 (local address 127.22.37.62:46099)
0620 18:36:22.151110 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:22.151114 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:22.151949 (+ 835us) server_negotiation.cc:407] Connection header received
0620 18:36:22.152110 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:22.152117 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:22.152211 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:22.152325 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:22.153305 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:22.154257 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:22.155928 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:22.156734 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:22.156874 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:22.160172 (+ 3298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:22.160177 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:22.160180 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:22.160209 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:22.163010 (+ 2801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:22.163676 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:22.163684 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:22.163688 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:22.163819 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:22.164177 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:22.164183 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:22.164187 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:22.164455 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:22.164605 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:22.165096 (+ 491us) server_negotiation.cc:299] Negotiation successful
0620 18:36:22.165279 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 18:36:22.165704 29538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:22.150842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:46099 (local address 127.0.0.1:43840)
0620 18:36:22.151805 (+ 963us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:22.151845 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:22.151981 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:22.152394 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:22.152398 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:22.152941 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:22.153152 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:22.153160 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:22.154402 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:22.154410 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:22.155779 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:22.155791 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:22.156937 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:22.156948 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:22.157101 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:22.157715 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:22.157731 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:22.159989 (+ 2258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:36:22.163132 (+ 3143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:22.163144 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:22.163151 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:22.163559 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:22.163926 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:22.163932 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:22.163935 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:22.164069 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:22.164588 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:36:22.164596 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:36:22.164791 (+ 195us) client_negotiation.cc:769] Sending connection context
0620 18:36:22.164999 (+ 208us) client_negotiation.cc:240] Negotiation successful
0620 18:36:22.165274 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":746,"thread_start_us":624,"threads_started":1}
I20260620 18:36:22.166843 29534 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.37.61" port: 44853 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:36:22 dist-test-slave-5lgp krb5kdc[28766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980582, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.37.60@KRBTEST.COM for oryx/127.22.37.61@KRBTEST.COM
I20260620 18:36:22.178723 29538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:22.167536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.61:44853 (local address 127.0.0.1:41182)
0620 18:36:22.167734 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:22.167753 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:22.167862 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:22.168172 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:22.168176 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:22.168495 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:22.168612 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:22.168619 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:22.169650 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:22.169653 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:22.170576 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:22.170586 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:22.171193 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:22.171197 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:22.171322 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:22.171807 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:22.171822 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:22.173611 (+ 1789us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:36:22.176864 (+ 3253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:22.176867 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:22.176870 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:22.177224 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:22.177575 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:22.177578 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:22.177580 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:22.177696 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:22.178107 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:36:22.178110 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:36:22.178207 (+ 97us) client_negotiation.cc:769] Sending connection context
0620 18:36:22.178319 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 18:36:22.178519 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 18:36:22.178750 29474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:22.167572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41182 (local address 127.22.37.61:44853)
0620 18:36:22.167747 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:22.167754 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:22.167800 (+ 46us) server_negotiation.cc:407] Connection header received
0620 18:36:22.167985 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:22.167996 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:22.168054 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:22.168158 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:22.168785 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:22.169549 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:22.170686 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:22.171078 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:22.171165 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:22.173789 (+ 2624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:22.173793 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:22.173797 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:22.173825 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:22.176750 (+ 2925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:22.177316 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:22.177319 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:22.177321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:22.177487 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:22.177780 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:22.177783 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:22.177785 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:22.178021 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:22.178120 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:22.178392 (+ 272us) server_negotiation.cc:299] Negotiation successful
0620 18:36:22.178544 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 18:36:22.179723 29534 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.37.60" port: 44179 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:36:22 dist-test-slave-5lgp krb5kdc[28766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980582, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.37.60@KRBTEST.COM for oryx/127.22.37.60@KRBTEST.COM
I20260620 18:36:22.191516 29538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:22.180484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.60:44179 (local address 127.0.0.1:40504)
0620 18:36:22.180673 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:22.180693 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:22.180791 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:22.181279 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:22.181282 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:22.181650 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:22.181797 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:22.181804 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:22.182744 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:22.182747 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:22.183617 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:22.183633 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:22.184344 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:22.184348 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:22.184500 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:22.184940 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:22.184955 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:22.186774 (+ 1819us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:36:22.189649 (+ 2875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:22.189652 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:22.189654 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:22.189975 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:22.190260 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:22.190263 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:22.190265 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:22.190389 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:22.190829 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:36:22.190833 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:36:22.190961 (+ 128us) client_negotiation.cc:769] Sending connection context
0620 18:36:22.191063 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 18:36:22.191215 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 18:36:22.191648 29537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:22.180909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40504 (local address 127.22.37.60:44179)
0620 18:36:22.181070 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:22.181075 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:22.181088 (+ 13us) server_negotiation.cc:407] Connection header received
0620 18:36:22.181123 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:22.181126 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:22.181174 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:22.181245 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:22.181906 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:22.182643 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:22.183740 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:22.184220 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:22.184312 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:22.186914 (+ 2602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:22.186918 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:22.186922 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:22.186946 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:22.189523 (+ 2577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:22.190062 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:22.190066 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:22.190068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:22.190160 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:22.190480 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:22.190483 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:22.190485 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:22.190723 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:22.190839 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:22.191196 (+ 357us) server_negotiation.cc:299] Negotiation successful
0620 18:36:22.191356 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 18:36:22.196389 29534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3b608bf35ad4fa2ab340274d7920a6c: Bootstrap starting.
I20260620 18:36:22.199971 29534 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e3b608bf35ad4fa2ab340274d7920a6c: Neither blocks nor log segments found. Creating new log.
I20260620 18:36:22.200976 29534 log.cc:824] T 00000000000000000000000000000000 P e3b608bf35ad4fa2ab340274d7920a6c: Log is configured to *not* fsync() on all Append() calls
I20260620 18:36:22.203472 29534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3b608bf35ad4fa2ab340274d7920a6c: No bootstrap required, opened a new log
I20260620 18:36:22.210237 29534 raft_consensus.cc:348] T 00000000000000000000000000000000 P e3b608bf35ad4fa2ab340274d7920a6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8fe34680f3c4d388068736f074cb839" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 46099 } } peers { permanent_uuid: "b1f845ed42fb4dcd90063c45dedac043" member_type: VOTER last_known_addr { host: "127.22.37.61" port: 44853 } } peers { permanent_uuid: "e3b608bf35ad4fa2ab340274d7920a6c" member_type: VOTER last_known_addr { host: "127.22.37.60" port: 44179 } }
I20260620 18:36:22.210565 29534 raft_consensus.cc:374] T 00000000000000000000000000000000 P e3b608bf35ad4fa2ab340274d7920a6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:36:22.210702 29534 raft_consensus.cc:729] T 00000000000000000000000000000000 P e3b608bf35ad4fa2ab340274d7920a6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3b608bf35ad4fa2ab340274d7920a6c, State: Initialized, Role: FOLLOWER
I20260620 18:36:22.211239 29534 consensus_queue.cc:260] T 00000000000000000000000000000000 P e3b608bf35ad4fa2ab340274d7920a6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8fe34680f3c4d388068736f074cb839" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 46099 } } peers { permanent_uuid: "b1f845ed42fb4dcd90063c45dedac043" member_type: VOTER last_known_addr { host: "127.22.37.61" port: 44853 } } peers { permanent_uuid: "e3b608bf35ad4fa2ab340274d7920a6c" member_type: VOTER last_known_addr { host: "127.22.37.60" port: 44179 } }
I20260620 18:36:22.212088 29540 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3b608bf35ad4fa2ab340274d7920a6c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8fe34680f3c4d388068736f074cb839" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 46099 } } peers { permanent_uuid: "b1f845ed42fb4dcd90063c45dedac043" member_type: VOTER last_known_addr { host: "127.22.37.61" port: 44853 } } peers { permanent_uuid: "e3b608bf35ad4fa2ab340274d7920a6c" member_type: VOTER last_known_addr { host: "127.22.37.60" port: 44179 } } }
I20260620 18:36:22.212363 29540 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3b608bf35ad4fa2ab340274d7920a6c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:36:22.213034 29534 sys_catalog.cc:564] T 00000000000000000000000000000000 P e3b608bf35ad4fa2ab340274d7920a6c [sys.catalog]: configured and running, proceeding with master startup.
W20260620 18:36:22.222975 29551 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e3b608bf35ad4fa2ab340274d7920a6c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 18:36:22.223119 29551 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 18:36:22.226662 29375 master_runner.cc:428] Detected that this master e3b608bf35ad4fa2ab340274d7920a6c is joining an existing cluster
I20260620 18:36:22.226826 29375 master_runner.cc:432] Initiating AddMaster RPC to add 127.22.37.60:44179
I20260620 18:36:22.252717 29344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:22.231121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43848 (local address 127.22.37.62:46099)
0620 18:36:22.231275 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:22.231282 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:22.232480 (+ 1198us) server_negotiation.cc:407] Connection header received
0620 18:36:22.232687 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:22.232692 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:22.232762 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:22.232859 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:22.238555 (+ 5696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:22.239714 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:22.242123 (+ 2409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:22.242774 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:22.242869 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:22.245647 (+ 2778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:22.245653 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:22.245658 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:22.245697 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:22.249489 (+ 3792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:22.250329 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:22.250334 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:22.250338 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:22.250500 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:22.250962 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:22.250967 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:22.250970 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:22.251295 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:22.251429 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:22.252118 (+ 689us) server_negotiation.cc:299] Negotiation successful
0620 18:36:22.252335 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 18:36:22.253002 29537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:22.231496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40514 (local address 127.22.37.60:44179)
0620 18:36:22.231633 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:22.231637 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:22.233286 (+ 1649us) server_negotiation.cc:407] Connection header received
0620 18:36:22.233524 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:22.233530 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:22.233595 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:22.233663 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:22.238619 (+ 4956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:22.239764 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:22.242181 (+ 2417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:22.242872 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:22.243003 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:22.245819 (+ 2816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:22.245825 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:22.245829 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:22.245859 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:22.249718 (+ 3859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:22.250456 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:22.250460 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:22.250463 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:22.250600 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:22.250934 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:22.250957 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:22.250959 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:22.251347 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:22.251461 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:22.252352 (+ 891us) server_negotiation.cc:299] Negotiation successful
0620 18:36:22.252650 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":15,"server-negotiator.queue_time_us":44}
I20260620 18:36:22.253682 29558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:22.230693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:46099 (local address 127.0.0.1:43848)
0620 18:36:22.232377 (+ 1684us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:22.232391 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:22.232549 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:22.232875 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:22.232879 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:22.234414 (+ 1535us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:22.238366 (+ 3952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:22.238381 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:22.239892 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:22.239896 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:22.241947 (+ 2051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:22.241961 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:22.243148 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:22.243155 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:22.243324 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:22.244034 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:22.244067 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:22.245432 (+ 1365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:36:22.249685 (+ 4253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:22.249690 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:22.249693 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:22.250190 (+ 497us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:22.250645 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:22.250652 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:22.250655 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:22.250829 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:22.251501 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:36:22.251505 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:36:22.251815 (+ 310us) client_negotiation.cc:769] Sending connection context
0620 18:36:22.252893 (+ 1078us) client_negotiation.cc:240] Negotiation successful
0620 18:36:22.253171 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1594,"mutex_wait_us":109,"spinlock_wait_cycles":33664,"thread_start_us":1759,"threads_started":1}
I20260620 18:36:22.253961 29557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:22.231789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.60:44179 (local address 127.0.0.1:40514)
0620 18:36:22.233190 (+ 1401us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:22.233211 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:22.233344 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:22.233706 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:22.233713 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:22.235187 (+ 1474us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:22.238442 (+ 3255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:22.238455 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:22.239962 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:22.239967 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:22.242047 (+ 2080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:22.242060 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:22.243038 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:22.243044 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:22.243212 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:22.244119 (+ 907us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:22.244143 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:22.245653 (+ 1510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:36:22.249851 (+ 4198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:22.249856 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:22.249860 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:22.250345 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:22.250711 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:22.250714 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:22.250719 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:22.250843 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:22.251509 (+ 666us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:36:22.251513 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:36:22.251729 (+ 216us) client_negotiation.cc:769] Sending connection context
0620 18:36:22.252531 (+ 802us) client_negotiation.cc:240] Negotiation successful
0620 18:36:22.253473 (+ 942us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":844,"mutex_wait_us":41,"thread_start_us":365,"threads_started":1}
I20260620 18:36:22.258355 29474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:22.230636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41184 (local address 127.22.37.61:44853)
0620 18:36:22.230824 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:22.230831 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:22.237829 (+ 6998us) server_negotiation.cc:407] Connection header received
0620 18:36:22.237961 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:22.237964 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:22.238058 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:22.238150 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:22.239582 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:22.241042 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:22.243789 (+ 2747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:22.244449 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:22.244571 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:22.248277 (+ 3706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:22.248283 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:22.248288 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:22.248325 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:22.252059 (+ 3734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:22.253461 (+ 1402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:22.253467 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:22.253470 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:22.253619 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:22.256690 (+ 3071us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:22.256697 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:22.256704 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:22.257085 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:22.257252 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:22.257771 (+ 519us) server_negotiation.cc:299] Negotiation successful
0620 18:36:22.258026 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 18:36:22.259016 29559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:22.232702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.61:44853 (local address 127.0.0.1:41184)
0620 18:36:22.237725 (+ 5023us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:22.237743 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:22.237866 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:22.238355 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:22.238361 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:22.239054 (+ 693us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:22.239381 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:22.239391 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:22.241259 (+ 1868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:22.241264 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:22.243648 (+ 2384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:22.243667 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:22.244830 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:22.244836 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:22.245008 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:22.245649 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:22.245670 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:22.248088 (+ 2418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:36:22.252222 (+ 4134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:22.252226 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:22.252230 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:22.252751 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:22.256363 (+ 3612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:22.256369 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:22.256373 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:22.256558 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:22.257221 (+ 663us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:36:22.257226 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:36:22.257464 (+ 238us) client_negotiation.cc:769] Sending connection context
0620 18:36:22.257629 (+ 165us) client_negotiation.cc:240] Negotiation successful
0620 18:36:22.257922 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4520,"mutex_wait_us":25,"thread_start_us":4328,"threads_started":1}
Jun 20 18:36:22 dist-test-slave-5lgp krb5kdc[28766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980579, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.37.62@KRBTEST.COM for oryx/127.22.37.60@KRBTEST.COM
I20260620 18:36:22.267719 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 28872
I20260620 18:36:22.273614 29537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:22.259039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40526 (local address 127.22.37.60:44179)
0620 18:36:22.259197 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:22.259204 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:22.259223 (+ 19us) server_negotiation.cc:407] Connection header received
0620 18:36:22.259468 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:22.259474 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:22.259559 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:22.259656 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:22.260919 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:22.261836 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:22.263514 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:22.263936 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:22.264034 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:22.267651 (+ 3617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:22.267658 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:22.267665 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:22.267703 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:22.271141 (+ 3438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:22.271777 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:22.271782 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:22.271786 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:22.271908 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:22.272264 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:22.272269 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:22.272272 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:22.272534 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:22.272681 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:22.273036 (+ 355us) server_negotiation.cc:299] Negotiation successful
0620 18:36:22.273235 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 18:36:22.273638 29366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:22.258983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.60:44179 (local address 127.0.0.1:40526)
0620 18:36:22.259165 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:22.259184 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:22.259280 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:22.259709 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:22.259715 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:22.260125 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:22.260770 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:22.260784 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:22.261960 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:22.261966 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:22.263368 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:22.263382 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:22.264073 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:22.264079 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:22.264240 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:22.264770 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:22.264795 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:22.267459 (+ 2664us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:36:22.271283 (+ 3824us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:22.271289 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:22.271293 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:22.271655 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:22.272017 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:22.272025 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:22.272030 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:22.272153 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:22.272649 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:36:22.272653 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:36:22.272806 (+ 153us) client_negotiation.cc:769] Sending connection context
0620 18:36:22.272945 (+ 139us) client_negotiation.cc:240] Negotiation successful
0620 18:36:22.273222 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260620 18:36:22.274886 28811 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.22.37.60:44179: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "e3b608bf35ad4fa2ab340274d7920a6c"
member_type: NON_VOTER
last_known_addr {
host: "127.22.37.60"
port: 44179
}
attrs {
promote: true
}
}
dest_uuid: "c8fe34680f3c4d388068736f074cb839"
cas_config_opid_index: 6
I20260620 18:36:22.275547 28811 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8fe34680f3c4d388068736f074cb839 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c8fe34680f3c4d388068736f074cb839" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 46099 } } peers { permanent_uuid: "b1f845ed42fb4dcd90063c45dedac043" member_type: VOTER last_known_addr { host: "127.22.37.61" port: 44853 } attrs { promote: false } } peers { permanent_uuid: "e3b608bf35ad4fa2ab340274d7920a6c" member_type: NON_VOTER last_known_addr { host: "127.22.37.60" port: 44179 } attrs { promote: true } }
I20260620 18:36:22.277783 29426 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b1f845ed42fb4dcd90063c45dedac043 [term 1 FOLLOWER]: Refusing update from remote peer c8fe34680f3c4d388068736f074cb839: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260620 18:36:22.278043 29509 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e3b608bf35ad4fa2ab340274d7920a6c [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:36:22.278519 29371 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c8fe34680f3c4d388068736f074cb839 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1f845ed42fb4dcd90063c45dedac043" member_type: VOTER last_known_addr { host: "127.22.37.61" port: 44853 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260620 18:36:22.281011 29368 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c8fe34680f3c4d388068736f074cb839 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER e3b608bf35ad4fa2ab340274d7920a6c (127.22.37.60) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c8fe34680f3c4d388068736f074cb839" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 46099 } } peers { permanent_uuid: "b1f845ed42fb4dcd90063c45dedac043" member_type: VOTER last_known_addr { host: "127.22.37.61" port: 44853 } attrs { promote: false } } peers { permanent_uuid: "e3b608bf35ad4fa2ab340274d7920a6c" member_type: NON_VOTER last_known_addr { host: "127.22.37.60" port: 44179 } attrs { promote: true } } }
I20260620 18:36:22.281627 29426 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b1f845ed42fb4dcd90063c45dedac043 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER e3b608bf35ad4fa2ab340274d7920a6c (127.22.37.60) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c8fe34680f3c4d388068736f074cb839" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 46099 } } peers { permanent_uuid: "b1f845ed42fb4dcd90063c45dedac043" member_type: VOTER last_known_addr { host: "127.22.37.61" port: 44853 } attrs { promote: false } } peers { permanent_uuid: "e3b608bf35ad4fa2ab340274d7920a6c" member_type: NON_VOTER last_known_addr { host: "127.22.37.60" port: 44179 } attrs { promote: true } } }
Jun 20 18:36:22 dist-test-slave-5lgp krb5kdc[28766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980578, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.37.1@KRBTEST.COM
I20260620 18:36:22.285241 29368 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.22.37.60:44179
I20260620 18:36:22.285866 29375 master.cc:537] Master@127.22.37.60:44179 shutting down...
I20260620 18:36:22.285974 29509 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e3b608bf35ad4fa2ab340274d7920a6c [term 1 FOLLOWER]: Refusing update from remote peer c8fe34680f3c4d388068736f074cb839: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260620 18:36:22.285794 29368 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8fe34680f3c4d388068736f074cb839 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c8fe34680f3c4d388068736f074cb839" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c8fe34680f3c4d388068736f074cb839" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 46099 } } peers { permanent_uuid: "b1f845ed42fb4dcd90063c45dedac043" member_type: VOTER last_known_addr { host: "127.22.37.61" port: 44853 } attrs { promote: false } } peers { permanent_uuid: "e3b608bf35ad4fa2ab340274d7920a6c" member_type: NON_VOTER last_known_addr { host: "127.22.37.60" port: 44179 } attrs { promote: true } } }
I20260620 18:36:22.286262 29368 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8fe34680f3c4d388068736f074cb839 [sys.catalog]: This master's current role is: LEADER
I20260620 18:36:22.287335 29369 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8fe34680f3c4d388068736f074cb839 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c8fe34680f3c4d388068736f074cb839" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c8fe34680f3c4d388068736f074cb839" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 46099 } } peers { permanent_uuid: "b1f845ed42fb4dcd90063c45dedac043" member_type: VOTER last_known_addr { host: "127.22.37.61" port: 44853 } attrs { promote: false } } peers { permanent_uuid: "e3b608bf35ad4fa2ab340274d7920a6c" member_type: NON_VOTER last_known_addr { host: "127.22.37.60" port: 44179 } attrs { promote: true } } }
I20260620 18:36:22.287599 29369 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8fe34680f3c4d388068736f074cb839 [sys.catalog]: This master's current role is: LEADER
I20260620 18:36:22.289799 29371 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c8fe34680f3c4d388068736f074cb839 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3b608bf35ad4fa2ab340274d7920a6c" member_type: NON_VOTER last_known_addr { host: "127.22.37.60" port: 44179 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 18:36:22.292670 28797 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260620 18:36:22.295818 29560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:22.269117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36616 (local address 127.22.37.1:35961)
0620 18:36:22.269856 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:22.269865 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:22.269915 (+ 50us) server_negotiation.cc:407] Connection header received
0620 18:36:22.270011 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:22.270022 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:22.270312 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:22.270490 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:22.272108 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:22.274545 (+ 2437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:22.276508 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:22.277413 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:22.278151 (+ 738us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:22.287501 (+ 9350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:22.287524 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:22.287542 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:22.287691 (+ 149us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:22.291093 (+ 3402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:22.291807 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:22.291819 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:22.291823 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:22.291979 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:22.292440 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:22.292444 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:22.292447 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:22.292894 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:22.294539 (+ 1645us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:22.294746 (+ 207us) server_negotiation.cc:299] Negotiation successful
0620 18:36:22.295177 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":383,"threads_started":1}
I20260620 18:36:22.297513 28995 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:36:22.299721 29454 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1f845ed42fb4dcd90063c45dedac043 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c8fe34680f3c4d388068736f074cb839" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c8fe34680f3c4d388068736f074cb839" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 46099 } } peers { permanent_uuid: "b1f845ed42fb4dcd90063c45dedac043" member_type: VOTER last_known_addr { host: "127.22.37.61" port: 44853 } attrs { promote: false } } peers { permanent_uuid: "e3b608bf35ad4fa2ab340274d7920a6c" member_type: NON_VOTER last_known_addr { host: "127.22.37.60" port: 44179 } attrs { promote: true } } }
I20260620 18:36:22.300021 29454 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1f845ed42fb4dcd90063c45dedac043 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 18:36:22.302410 28797 consensus_peers.cc:487] T 00000000000000000000000000000000 P c8fe34680f3c4d388068736f074cb839 -> Peer e3b608bf35ad4fa2ab340274d7920a6c (127.22.37.60:44179): Couldn't send request to peer e3b608bf35ad4fa2ab340274d7920a6c. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 18:36:22.315409 29375 raft_consensus.cc:2229] T 00000000000000000000000000000000 P e3b608bf35ad4fa2ab340274d7920a6c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:36:22.315879 29375 raft_consensus.cc:2258] T 00000000000000000000000000000000 P e3b608bf35ad4fa2ab340274d7920a6c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:36:22.316112 29375 tablet_replica.cc:330] T 00000000000000000000000000000000 P e3b608bf35ad4fa2ab340274d7920a6c: stopping tablet replica
I20260620 18:36:22.321867 29375 master.cc:552] Master@127.22.37.60:44179 shutdown complete.
I20260620 18:36:22.321956 29375 master_runner.cc:305] Clearing existing system tablet
I20260620 18:36:22.322299 29375 fs_manager.cc:359] Metadata directory not provided
I20260620 18:36:22.322422 29375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:36:22.324599 29375 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 18:36:22.326033 29564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:22.326375 29375 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:36:22.326651 29375 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:36:22.326754 29375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/master-2/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/master-2/wal
uuid: "e3b608bf35ad4fa2ab340274d7920a6c"
format_stamp: "Formatted at 2026-06-20 18:36:22 on dist-test-slave-5lgp"
server_key: "6beb82d6f924561048c1fbb60fe5770e"
server_key_iv: "cc8349e51006b0a33fb931cd3902f918"
server_key_version: "encryptionkey@0"
I20260620 18:36:22.330819 29375 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P e3b608bf35ad4fa2ab340274d7920a6c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:36:22.336339 29375 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P e3b608bf35ad4fa2ab340274d7920a6c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 18:36:22.336495 29375 log.cc:1196] T 00000000000000000000000000000000 P e3b608bf35ad4fa2ab340274d7920a6c: Deleting WAL directory at /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260620 18:36:22.336899 29375 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P e3b608bf35ad4fa2ab340274d7920a6c: Deleting consensus metadata
I20260620 18:36:22.337139 29375 master_runner.cc:315] Copying system tablet from 127.22.37.62:46099
I20260620 18:36:22.339809 29375 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P e3b608bf35ad4fa2ab340274d7920a6c: tablet copy: Beginning tablet copy session from remote peer at address 127.22.37.62:46099
I20260620 18:36:22.352694 29344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:22.340546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43862 (local address 127.22.37.62:46099)
0620 18:36:22.340806 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:22.340815 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:22.341301 (+ 486us) server_negotiation.cc:407] Connection header received
0620 18:36:22.341522 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:22.341528 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:22.341614 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:22.341725 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:22.342578 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:22.343404 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:22.344621 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:22.345024 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:22.345154 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:22.347092 (+ 1938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:22.347099 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:22.347103 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:22.347142 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:22.350199 (+ 3057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:22.350802 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:22.350810 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:22.350813 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:22.350948 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:22.351347 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:22.351352 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:22.351355 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:22.351630 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:22.351790 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:22.352131 (+ 341us) server_negotiation.cc:299] Negotiation successful
0620 18:36:22.352332 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260620 18:36:22.352728 29570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:22.340522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:46099 (local address 127.0.0.1:43862)
0620 18:36:22.341193 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:22.341208 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:22.341367 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:22.341869 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:22.341873 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:22.342247 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:22.342426 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:22.342434 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:22.343554 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:22.343559 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:22.344462 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:22.344472 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:22.345180 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:22.345185 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:22.345358 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:22.346090 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:22.346107 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:22.346887 (+ 780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:36:22.350332 (+ 3445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:22.350340 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:22.350344 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:22.350685 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:22.351102 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:22.351105 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:22.351107 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:22.351231 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:22.351745 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:36:22.351751 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:36:22.351899 (+ 148us) client_negotiation.cc:769] Sending connection context
0620 18:36:22.352046 (+ 147us) client_negotiation.cc:240] Negotiation successful
0620 18:36:22.352327 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":441,"threads_started":1}
I20260620 18:36:22.353762 28831 tablet_copy_service.cc:140] P c8fe34680f3c4d388068736f074cb839: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer e3b608bf35ad4fa2ab340274d7920a6c ({username='oryx', principal='oryx/127.22.37.60@KRBTEST.COM'} at 127.0.0.1:43862)
I20260620 18:36:22.353935 28831 tablet_copy_service.cc:161] P c8fe34680f3c4d388068736f074cb839: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer e3b608bf35ad4fa2ab340274d7920a6c at {username='oryx', principal='oryx/127.22.37.60@KRBTEST.COM'} at 127.0.0.1:43862: session id = e3b608bf35ad4fa2ab340274d7920a6c-00000000000000000000000000000000
I20260620 18:36:22.358058 28831 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P c8fe34680f3c4d388068736f074cb839: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 18:36:22.359856 29375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 18:36:22.366456 29375 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P e3b608bf35ad4fa2ab340274d7920a6c: tablet copy: Starting download of 0 data blocks...
I20260620 18:36:22.366737 29375 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P e3b608bf35ad4fa2ab340274d7920a6c: tablet copy: Starting download of 1 WAL segments...
I20260620 18:36:22.369598 29375 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P e3b608bf35ad4fa2ab340274d7920a6c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 18:36:22.372604 28831 tablet_copy_service.cc:342] P c8fe34680f3c4d388068736f074cb839: Request end of tablet copy session e3b608bf35ad4fa2ab340274d7920a6c-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.22.37.60@KRBTEST.COM'} at 127.0.0.1:43862
I20260620 18:36:22.372745 28831 tablet_copy_service.cc:433] P c8fe34680f3c4d388068736f074cb839: ending tablet copy session e3b608bf35ad4fa2ab340274d7920a6c-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer e3b608bf35ad4fa2ab340274d7920a6c
I20260620 18:36:22.382896 29375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:36:22.385558 29571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:22.390040 29572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:22.397102 29574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:22.397406 29375 server_base.cc:1034] running on GCE node
I20260620 18:36:22.397805 29375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:36:22.398386 29375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:36:22.399634 29375 hybrid_clock.cc:648] HybridClock initialized: now 1781980582399604 us; error 25 us; skew 500 ppm
I20260620 18:36:22.399852 29375 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:36:22 dist-test-slave-5lgp krb5kdc[28766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980582, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.37.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:36:22.401803 29375 init.cc:377] Logged in from keytab as oryx/127.22.37.60@KRBTEST.COM (short username oryx)
I20260620 18:36:22.403131 29375 webserver.cc:460] Webserver started at http://127.22.37.60:39501/ using document root <none> and password file <none>
I20260620 18:36:22.403431 29375 fs_manager.cc:359] Metadata directory not provided
I20260620 18:36:22.403543 29375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:36:22.405761 29375 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.002s
I20260620 18:36:22.407728 29581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:22.408082 29375 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:36:22.408196 29375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/master-2/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/master-2/wal
uuid: "e3b608bf35ad4fa2ab340274d7920a6c"
format_stamp: "Formatted at 2026-06-20 18:36:22 on dist-test-slave-5lgp"
server_key: "6beb82d6f924561048c1fbb60fe5770e"
server_key_iv: "cc8349e51006b0a33fb931cd3902f918"
server_key_version: "encryptionkey@0"
I20260620 18:36:22.408401 29375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:36:22.438481 29375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:36:22.442138 29375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:36:22.456497 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 28872
I20260620 18:36:22.480955 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.37.1:35961
--local_ip_for_outbound_sockets=127.22.37.1
--tserver_master_addrs=127.22.37.62:46099
--webserver_port=42569
--webserver_interface=127.22.37.1
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/krb5kdc/oryx_127.22.37.1.keytab
--principal=oryx/127.22.37.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:38175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.22.37.62:46099,127.22.37.61:44853
--tserver_master_addrs=127.22.37.62:46099,127.22.37.61:44853,127.22.37.60:44179 with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:36:22.493472 29375 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.60:44179
I20260620 18:36:22.493472 29633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.60:44179 every 8 connection(s)
I20260620 18:36:22.494421 29375 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/master-2/data/info.pb
I20260620 18:36:22.501281 29634 sys_catalog.cc:263] Verifying existing consensus state
I20260620 18:36:22.506639 29634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3b608bf35ad4fa2ab340274d7920a6c: Bootstrap starting.
I20260620 18:36:22.521711 29634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3b608bf35ad4fa2ab340274d7920a6c: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 18:36:22.522172 29634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3b608bf35ad4fa2ab340274d7920a6c: Bootstrap complete.
I20260620 18:36:22.523075 29634 raft_consensus.cc:348] T 00000000000000000000000000000000 P e3b608bf35ad4fa2ab340274d7920a6c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c8fe34680f3c4d388068736f074cb839" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 46099 } } peers { permanent_uuid: "b1f845ed42fb4dcd90063c45dedac043" member_type: VOTER last_known_addr { host: "127.22.37.61" port: 44853 } attrs { promote: false } } peers { permanent_uuid: "e3b608bf35ad4fa2ab340274d7920a6c" member_type: NON_VOTER last_known_addr { host: "127.22.37.60" port: 44179 } attrs { promote: true } }
I20260620 18:36:22.523320 29634 raft_consensus.cc:729] T 00000000000000000000000000000000 P e3b608bf35ad4fa2ab340274d7920a6c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e3b608bf35ad4fa2ab340274d7920a6c, State: Initialized, Role: LEARNER
I20260620 18:36:22.523546 29634 consensus_queue.cc:260] T 00000000000000000000000000000000 P e3b608bf35ad4fa2ab340274d7920a6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c8fe34680f3c4d388068736f074cb839" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 46099 } } peers { permanent_uuid: "b1f845ed42fb4dcd90063c45dedac043" member_type: VOTER last_known_addr { host: "127.22.37.61" port: 44853 } attrs { promote: false } } peers { permanent_uuid: "e3b608bf35ad4fa2ab340274d7920a6c" member_type: NON_VOTER last_known_addr { host: "127.22.37.60" port: 44179 } attrs { promote: true } }
I20260620 18:36:22.524367 29637 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3b608bf35ad4fa2ab340274d7920a6c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c8fe34680f3c4d388068736f074cb839" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 46099 } } peers { permanent_uuid: "b1f845ed42fb4dcd90063c45dedac043" member_type: VOTER last_known_addr { host: "127.22.37.61" port: 44853 } attrs { promote: false } } peers { permanent_uuid: "e3b608bf35ad4fa2ab340274d7920a6c" member_type: NON_VOTER last_known_addr { host: "127.22.37.60" port: 44179 } attrs { promote: true } } }
I20260620 18:36:22.524729 29634 sys_catalog.cc:564] T 00000000000000000000000000000000 P e3b608bf35ad4fa2ab340274d7920a6c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:36:22.524767 29637 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3b608bf35ad4fa2ab340274d7920a6c [sys.catalog]: This master's current role is: LEARNER
I20260620 18:36:22.533571 29648 catalog_manager.cc:1259] Loaded cluster ID: eee720b350864e718f512e0ff17d1a2e
I20260620 18:36:22.533674 29648 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e3b608bf35ad4fa2ab340274d7920a6c: loading cluster ID for follower catalog manager: success
I20260620 18:36:22.536198 29648 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e3b608bf35ad4fa2ab340274d7920a6c: acquiring CA information for follower catalog manager: success
I20260620 18:36:22.538014 29648 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e3b608bf35ad4fa2ab340274d7920a6c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 18:36:22.712656 29366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:22.704417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.60:44179 (local address 127.0.0.1:40542)
0620 18:36:22.704683 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:22.704706 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:22.704849 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:22.705888 (+ 1039us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:22.705895 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:22.705937 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:36:22.706154 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:22.706165 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:22.708142 (+ 1977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:22.708148 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:22.710154 (+ 2006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:22.710170 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:22.711847 (+ 1677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:22.711853 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:22.712075 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:22.712112 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 18:36:22.712212 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 18:36:22.712352 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143}
I20260620 18:36:22.712738 29649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:22.704715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40542 (local address 127.22.37.60:44179)
0620 18:36:22.705491 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:22.705499 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:22.705525 (+ 26us) server_negotiation.cc:407] Connection header received
0620 18:36:22.705609 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:22.705614 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:22.705694 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:22.705814 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:36:22.706343 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:22.707961 (+ 1618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:22.710311 (+ 2350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:22.711657 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:22.711783 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:22.712097 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:22.712253 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 18:36:22.712392 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":424,"threads_started":1}
I20260620 18:36:22.713948 29608 raft_consensus.cc:1203] T 00000000000000000000000000000000 P e3b608bf35ad4fa2ab340274d7920a6c [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.7] Dedup: 1.7->[]
I20260620 18:36:22.714505 29637 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3b608bf35ad4fa2ab340274d7920a6c [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8fe34680f3c4d388068736f074cb839. Latest consensus state: current_term: 1 leader_uuid: "c8fe34680f3c4d388068736f074cb839" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c8fe34680f3c4d388068736f074cb839" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 46099 } } peers { permanent_uuid: "b1f845ed42fb4dcd90063c45dedac043" member_type: VOTER last_known_addr { host: "127.22.37.61" port: 44853 } attrs { promote: false } } peers { permanent_uuid: "e3b608bf35ad4fa2ab340274d7920a6c" member_type: NON_VOTER last_known_addr { host: "127.22.37.60" port: 44179 } attrs { promote: true } } }
I20260620 18:36:22.714805 29637 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3b608bf35ad4fa2ab340274d7920a6c [sys.catalog]: This master's current role is: LEARNER
W20260620 18:36:22.850993 29624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:36:22.851265 29624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:36:22.851461 29624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:36:22.858956 29624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:36:22.859062 29624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:36:22.859244 29624 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.37.1
I20260620 18:36:22.869257 29624 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:38175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/krb5kdc/oryx_127.22.37.1.keytab
--principal=oryx/127.22.37.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.37.1:35961
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.37.1
--webserver_port=42569
--tserver_master_addrs=127.22.37.62:46099,127.22.37.61:44853,127.22.37.60:44179
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.37.1
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:36:22.870698 29624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:36:22.871986 29624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:36:22.886515 29655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:22.886917 29657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:22.887825 29654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:22.889212 29624 server_base.cc:1034] running on GCE node
I20260620 18:36:22.889834 29624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:36:22.890743 29624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:36:22.891969 29624 hybrid_clock.cc:648] HybridClock initialized: now 1781980582891939 us; error 61 us; skew 500 ppm
I20260620 18:36:22.892282 29624 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:36:22 dist-test-slave-5lgp krb5kdc[28766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980582, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.37.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:36:22.895916 29624 init.cc:377] Logged in from keytab as oryx/127.22.37.1@KRBTEST.COM (short username oryx)
I20260620 18:36:22.897604 29624 webserver.cc:460] Webserver started at http://127.22.37.1:42569/ using document root <none> and password file <none>
I20260620 18:36:22.898311 29624 fs_manager.cc:359] Metadata directory not provided
I20260620 18:36:22.898458 29624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:36:22.903792 29624 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260620 18:36:22.906509 29664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:22.907856 29624 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 18:36:22.908028 29624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/ts-0/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/ts-0/wal
uuid: "7232cd23c889418689f2c886ba008313"
format_stamp: "Formatted at 2026-06-20 18:36:19 on dist-test-slave-5lgp"
server_key: "c7e09bd1cd6e2bbc61456e599776464e"
server_key_iv: "700bfd76d7261ab63464ff09a27da685"
server_key_version: "encryptionkey@0"
I20260620 18:36:22.909020 29624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:36:22.928041 29624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:36:22.932431 29624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:36:22.932657 29624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:36:22.933696 29624 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:36:22.934935 29624 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:36:22.935019 29624 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:22.935106 29624 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:36:22.935159 29624 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:22.971084 29624 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.1:35961
I20260620 18:36:22.971125 29776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.1:35961 every 8 connection(s)
I20260620 18:36:22.972574 29624 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/ts-0/data/info.pb
I20260620 18:36:22.980808 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 29624
I20260620 18:36:22.981163 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 29006
Jun 20 18:36:22 dist-test-slave-5lgp krb5kdc[28766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980582, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.37.1@KRBTEST.COM for oryx/127.22.37.61@KRBTEST.COM
Jun 20 18:36:22 dist-test-slave-5lgp krb5kdc[28766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980582, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.37.1@KRBTEST.COM for oryx/127.22.37.60@KRBTEST.COM
Jun 20 18:36:22 dist-test-slave-5lgp krb5kdc[28766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980582, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.37.1@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM
I20260620 18:36:22.999825 29782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:22.975876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.1:41115 (local address 127.22.37.61:44853)
0620 18:36:22.976390 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:22.976399 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:22.978260 (+ 1861us) server_negotiation.cc:407] Connection header received
0620 18:36:22.980621 (+ 2361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:22.980626 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:22.980705 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:22.980787 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:22.984639 (+ 3852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:22.985543 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:22.986992 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:22.987439 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:22.987554 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:22.992979 (+ 5425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:22.992986 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:22.992991 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:22.993031 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:22.996629 (+ 3598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:22.997522 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:22.997527 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:22.997531 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:22.997668 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:22.998106 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:22.998111 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:22.998114 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:22.998378 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:22.998529 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:22.999220 (+ 691us) server_negotiation.cc:299] Negotiation successful
0620 18:36:22.999430 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":260,"threads_started":1}
I20260620 18:36:23.001508 29785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:22.976193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.61:44853 (local address 127.22.37.1:41115)
0620 18:36:22.978138 (+ 1945us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:22.978157 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:22.980308 (+ 2151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:22.981075 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:22.981087 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:22.982164 (+ 1077us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:22.983243 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:22.983271 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:22.985686 (+ 2415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:22.985691 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:22.986834 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:22.986846 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:22.987878 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:22.987885 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:22.988216 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:22.989507 (+ 1291us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:22.989543 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:22.992742 (+ 3199us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:36:22.996800 (+ 4058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:22.996810 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:22.996827 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:22.997417 (+ 590us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:22.997780 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:22.997786 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:22.997789 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:22.997982 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:22.998503 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:36:22.998511 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:36:22.998843 (+ 332us) client_negotiation.cc:769] Sending connection context
0620 18:36:22.999107 (+ 264us) client_negotiation.cc:240] Negotiation successful
0620 18:36:22.999672 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1780,"mutex_wait_us":298,"spinlock_wait_cycles":73216,"thread_start_us":666,"threads_started":1}
I20260620 18:36:23.003252 29781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:22.975537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.1:48103 (local address 127.22.37.62:46099)
0620 18:36:22.976960 (+ 1423us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:22.976968 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:22.978314 (+ 1346us) server_negotiation.cc:407] Connection header received
0620 18:36:22.980619 (+ 2305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:22.980625 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:22.980705 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:22.980810 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:22.983840 (+ 3030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:22.984896 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:22.989975 (+ 5079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:22.990569 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:22.990715 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:22.995552 (+ 4837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:22.995560 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:22.995566 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:22.995610 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:22.999227 (+ 3617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:23.000355 (+ 1128us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:23.000361 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:23.000364 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:23.000955 (+ 591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:23.001338 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:23.001343 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:23.001346 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:23.002162 (+ 816us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:23.002578 (+ 416us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:23.002716 (+ 138us) server_negotiation.cc:299] Negotiation successful
0620 18:36:23.002917 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1284,"thread_start_us":1182,"threads_started":1}
I20260620 18:36:23.003268 29783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:22.976812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:46099 (local address 127.22.37.1:48103)
0620 18:36:22.978217 (+ 1405us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:22.978232 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:22.980313 (+ 2081us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:22.981077 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:22.981088 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:22.982175 (+ 1087us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:22.983682 (+ 1507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:22.983697 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:22.987496 (+ 3799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:22.987503 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:22.989709 (+ 2206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:22.989731 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:22.990801 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:22.990814 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:22.991048 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:22.991646 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:22.991672 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:22.995140 (+ 3468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:36:22.999384 (+ 4244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:22.999388 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:22.999393 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:23.000216 (+ 823us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:23.001064 (+ 848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:23.001069 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:23.001073 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:23.001205 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:23.002276 (+ 1071us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:36:23.002282 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:36:23.002454 (+ 172us) client_negotiation.cc:769] Sending connection context
0620 18:36:23.002572 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 18:36:23.002847 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1293,"mutex_wait_us":120,"spinlock_wait_cycles":64640,"thread_start_us":380,"threads_started":1}
I20260620 18:36:23.003628 29649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:22.975646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.1:54545 (local address 127.22.37.60:44179)
0620 18:36:22.975828 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:22.975836 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:22.978073 (+ 2237us) server_negotiation.cc:407] Connection header received
0620 18:36:22.981115 (+ 3042us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:22.981120 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:22.981196 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:22.981266 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:22.983437 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:22.984451 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:22.987442 (+ 2991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:22.988052 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:22.988174 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:22.994676 (+ 6502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:22.994682 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:22.994688 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:22.994731 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:22.998835 (+ 4104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:23.001483 (+ 2648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:23.001488 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:23.001491 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:23.001632 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:23.002105 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:23.002110 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:23.002113 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:23.002440 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:23.002569 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:23.003060 (+ 491us) server_negotiation.cc:299] Negotiation successful
0620 18:36:23.003286 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 18:36:23.003825 29779 heartbeater.cc:347] Connected to a master server at 127.22.37.61:44853
I20260620 18:36:23.004043 29778 heartbeater.cc:347] Connected to a master server at 127.22.37.62:46099
I20260620 18:36:23.004166 29778 heartbeater.cc:464] Registering TS with master...
I20260620 18:36:23.004170 29779 heartbeater.cc:464] Registering TS with master...
I20260620 18:36:23.004990 29784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:22.976193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.60:44179 (local address 127.22.37.1:54545)
0620 18:36:22.977873 (+ 1680us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:22.977924 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:22.980308 (+ 2384us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:22.981558 (+ 1250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:22.981562 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:22.982224 (+ 662us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:22.983240 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:22.983268 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:22.984649 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:22.984657 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:22.987278 (+ 2621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:22.987297 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:22.988843 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:22.988854 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:22.989542 (+ 688us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:22.991166 (+ 1624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:22.991183 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:22.994441 (+ 3258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:36:23.000730 (+ 6289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:23.000736 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:23.000740 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:23.001341 (+ 601us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:23.001774 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:23.001779 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:23.001783 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:23.001963 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:23.002638 (+ 675us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:36:23.002642 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:36:23.002757 (+ 115us) client_negotiation.cc:769] Sending connection context
0620 18:36:23.004278 (+ 1521us) client_negotiation.cc:240] Negotiation successful
0620 18:36:23.004543 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":932,"mutex_wait_us":76,"spinlock_wait_cycles":121600,"thread_start_us":1631,"threads_started":1}
I20260620 18:36:23.005141 29778 heartbeater.cc:511] Master 127.22.37.62:46099 requested a full tablet report, sending...
I20260620 18:36:23.005164 29779 heartbeater.cc:511] Master 127.22.37.61:44853 requested a full tablet report, sending...
I20260620 18:36:23.006479 29777 heartbeater.cc:347] Connected to a master server at 127.22.37.60:44179
I20260620 18:36:23.006620 29777 heartbeater.cc:464] Registering TS with master...
I20260620 18:36:23.007004 29777 heartbeater.cc:511] Master 127.22.37.60:44179 requested a full tablet report, sending...
I20260620 18:36:23.007442 28811 ts_manager.cc:194] Re-registered known tserver with Master: 7232cd23c889418689f2c886ba008313 (127.22.37.1:35961)
I20260620 18:36:23.007682 29416 ts_manager.cc:194] Registered new tserver with Master: 7232cd23c889418689f2c886ba008313 (127.22.37.1:35961)
I20260620 18:36:23.008574 28811 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.37.1@KRBTEST.COM'} at 127.22.37.1:48103
I20260620 18:36:23.010210 29598 ts_manager.cc:194] Registered new tserver with Master: 7232cd23c889418689f2c886ba008313 (127.22.37.1:35961)
Jun 20 18:36:23 dist-test-slave-5lgp krb5kdc[28766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980578, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.37.2@KRBTEST.COM
I20260620 18:36:23.018952 29787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:22.982581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56066 (local address 127.22.37.2:33791)
0620 18:36:22.983260 (+ 679us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:22.983268 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:22.994230 (+ 10962us) server_negotiation.cc:407] Connection header received
0620 18:36:22.994315 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:22.994326 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:22.994678 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:22.994838 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:22.998993 (+ 4155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:23.000554 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:23.005725 (+ 5171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:23.006326 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:23.007308 (+ 982us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:23.013011 (+ 5703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:23.013024 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:23.013037 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:23.013074 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:23.016367 (+ 3293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:23.016997 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:23.017004 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:23.017006 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:23.017131 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:23.017526 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:23.017530 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:23.017532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:23.017873 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:23.018104 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:23.018362 (+ 258us) server_negotiation.cc:299] Negotiation successful
0620 18:36:23.018546 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":347,"threads_started":1}
I20260620 18:36:23.020140 29129 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:36:23.095990 29371 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8fe34680f3c4d388068736f074cb839 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c8fe34680f3c4d388068736f074cb839" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c8fe34680f3c4d388068736f074cb839" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 46099 } } peers { permanent_uuid: "b1f845ed42fb4dcd90063c45dedac043" member_type: VOTER last_known_addr { host: "127.22.37.61" port: 44853 } attrs { promote: false } } peers { permanent_uuid: "e3b608bf35ad4fa2ab340274d7920a6c" member_type: NON_VOTER last_known_addr { host: "127.22.37.60" port: 44179 } attrs { promote: true } } }
I20260620 18:36:23.096416 29371 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8fe34680f3c4d388068736f074cb839 [sys.catalog]: This master's current role is: LEADER
I20260620 18:36:23.097724 29790 raft_consensus.cc:1050] T 00000000000000000000000000000000 P c8fe34680f3c4d388068736f074cb839: attempting to promote NON_VOTER e3b608bf35ad4fa2ab340274d7920a6c to VOTER
I20260620 18:36:23.098351 29790 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8fe34680f3c4d388068736f074cb839 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c8fe34680f3c4d388068736f074cb839" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 46099 } } peers { permanent_uuid: "b1f845ed42fb4dcd90063c45dedac043" member_type: VOTER last_known_addr { host: "127.22.37.61" port: 44853 } attrs { promote: false } } peers { permanent_uuid: "e3b608bf35ad4fa2ab340274d7920a6c" member_type: VOTER last_known_addr { host: "127.22.37.60" port: 44179 } attrs { promote: false } }
I20260620 18:36:23.100049 29608 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e3b608bf35ad4fa2ab340274d7920a6c [term 1 LEARNER]: Refusing update from remote peer c8fe34680f3c4d388068736f074cb839: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260620 18:36:23.100124 29426 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b1f845ed42fb4dcd90063c45dedac043 [term 1 FOLLOWER]: Refusing update from remote peer c8fe34680f3c4d388068736f074cb839: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260620 18:36:23.100628 29789 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c8fe34680f3c4d388068736f074cb839 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3b608bf35ad4fa2ab340274d7920a6c" member_type: VOTER last_known_addr { host: "127.22.37.60" port: 44179 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260620 18:36:23.101020 29371 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c8fe34680f3c4d388068736f074cb839 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1f845ed42fb4dcd90063c45dedac043" member_type: VOTER last_known_addr { host: "127.22.37.61" port: 44853 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260620 18:36:23.103498 29371 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c8fe34680f3c4d388068736f074cb839 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e3b608bf35ad4fa2ab340274d7920a6c (127.22.37.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c8fe34680f3c4d388068736f074cb839" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 46099 } } peers { permanent_uuid: "b1f845ed42fb4dcd90063c45dedac043" member_type: VOTER last_known_addr { host: "127.22.37.61" port: 44853 } attrs { promote: false } } peers { permanent_uuid: "e3b608bf35ad4fa2ab340274d7920a6c" member_type: VOTER last_known_addr { host: "127.22.37.60" port: 44179 } attrs { promote: false } } }
I20260620 18:36:23.105810 29608 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e3b608bf35ad4fa2ab340274d7920a6c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e3b608bf35ad4fa2ab340274d7920a6c (127.22.37.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c8fe34680f3c4d388068736f074cb839" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 46099 } } peers { permanent_uuid: "b1f845ed42fb4dcd90063c45dedac043" member_type: VOTER last_known_addr { host: "127.22.37.61" port: 44853 } attrs { promote: false } } peers { permanent_uuid: "e3b608bf35ad4fa2ab340274d7920a6c" member_type: VOTER last_known_addr { host: "127.22.37.60" port: 44179 } attrs { promote: false } } }
I20260620 18:36:23.106545 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 29006
I20260620 18:36:23.108328 29371 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8fe34680f3c4d388068736f074cb839 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c8fe34680f3c4d388068736f074cb839" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c8fe34680f3c4d388068736f074cb839" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 46099 } } peers { permanent_uuid: "b1f845ed42fb4dcd90063c45dedac043" member_type: VOTER last_known_addr { host: "127.22.37.61" port: 44853 } attrs { promote: false } } peers { permanent_uuid: "e3b608bf35ad4fa2ab340274d7920a6c" member_type: VOTER last_known_addr { host: "127.22.37.60" port: 44179 } attrs { promote: false } } }
I20260620 18:36:23.108579 29371 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8fe34680f3c4d388068736f074cb839 [sys.catalog]: This master's current role is: LEADER
I20260620 18:36:23.108811 29371 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8fe34680f3c4d388068736f074cb839 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c8fe34680f3c4d388068736f074cb839" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c8fe34680f3c4d388068736f074cb839" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 46099 } } peers { permanent_uuid: "b1f845ed42fb4dcd90063c45dedac043" member_type: VOTER last_known_addr { host: "127.22.37.61" port: 44853 } attrs { promote: false } } peers { permanent_uuid: "e3b608bf35ad4fa2ab340274d7920a6c" member_type: VOTER last_known_addr { host: "127.22.37.60" port: 44179 } attrs { promote: false } } }
I20260620 18:36:23.109023 29371 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8fe34680f3c4d388068736f074cb839 [sys.catalog]: This master's current role is: LEADER
I20260620 18:36:23.109527 29789 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8fe34680f3c4d388068736f074cb839 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c8fe34680f3c4d388068736f074cb839" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c8fe34680f3c4d388068736f074cb839" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 46099 } } peers { permanent_uuid: "b1f845ed42fb4dcd90063c45dedac043" member_type: VOTER last_known_addr { host: "127.22.37.61" port: 44853 } attrs { promote: false } } peers { permanent_uuid: "e3b608bf35ad4fa2ab340274d7920a6c" member_type: VOTER last_known_addr { host: "127.22.37.60" port: 44179 } attrs { promote: false } } }
I20260620 18:36:23.109794 29789 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8fe34680f3c4d388068736f074cb839 [sys.catalog]: This master's current role is: LEADER
I20260620 18:36:23.110100 29426 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b1f845ed42fb4dcd90063c45dedac043 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e3b608bf35ad4fa2ab340274d7920a6c (127.22.37.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c8fe34680f3c4d388068736f074cb839" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 46099 } } peers { permanent_uuid: "b1f845ed42fb4dcd90063c45dedac043" member_type: VOTER last_known_addr { host: "127.22.37.61" port: 44853 } attrs { promote: false } } peers { permanent_uuid: "e3b608bf35ad4fa2ab340274d7920a6c" member_type: VOTER last_known_addr { host: "127.22.37.60" port: 44179 } attrs { promote: false } } }
I20260620 18:36:23.110965 29637 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3b608bf35ad4fa2ab340274d7920a6c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c8fe34680f3c4d388068736f074cb839" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c8fe34680f3c4d388068736f074cb839" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 46099 } } peers { permanent_uuid: "b1f845ed42fb4dcd90063c45dedac043" member_type: VOTER last_known_addr { host: "127.22.37.61" port: 44853 } attrs { promote: false } } peers { permanent_uuid: "e3b608bf35ad4fa2ab340274d7920a6c" member_type: VOTER last_known_addr { host: "127.22.37.60" port: 44179 } attrs { promote: false } } }
I20260620 18:36:23.111196 29637 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3b608bf35ad4fa2ab340274d7920a6c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:36:23.114648 29793 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1f845ed42fb4dcd90063c45dedac043 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c8fe34680f3c4d388068736f074cb839" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c8fe34680f3c4d388068736f074cb839" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 46099 } } peers { permanent_uuid: "b1f845ed42fb4dcd90063c45dedac043" member_type: VOTER last_known_addr { host: "127.22.37.61" port: 44853 } attrs { promote: false } } peers { permanent_uuid: "e3b608bf35ad4fa2ab340274d7920a6c" member_type: VOTER last_known_addr { host: "127.22.37.60" port: 44179 } attrs { promote: false } } }
I20260620 18:36:23.114959 29793 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1f845ed42fb4dcd90063c45dedac043 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:36:23.124826 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.37.2:33791
--local_ip_for_outbound_sockets=127.22.37.2
--tserver_master_addrs=127.22.37.62:46099
--webserver_port=40713
--webserver_interface=127.22.37.2
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/krb5kdc/oryx_127.22.37.2.keytab
--principal=oryx/127.22.37.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:38175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.22.37.62:46099,127.22.37.61:44853
--tserver_master_addrs=127.22.37.62:46099,127.22.37.61:44853,127.22.37.60:44179 with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:36:23.453666 29794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:36:23.453933 29794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:36:23.454098 29794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:36:23.461515 29794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:36:23.461608 29794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:36:23.461791 29794 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.37.2
I20260620 18:36:23.471364 29794 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:38175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/krb5kdc/oryx_127.22.37.2.keytab
--principal=oryx/127.22.37.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.37.2:33791
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.37.2
--webserver_port=40713
--tserver_master_addrs=127.22.37.62:46099,127.22.37.61:44853,127.22.37.60:44179
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.37.2
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:36:23.472599 29794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:36:23.474018 29794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:36:23.486634 29799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:23.486671 29800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:23.486703 29802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:23.487991 29794 server_base.cc:1034] running on GCE node
I20260620 18:36:23.488615 29794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:36:23.489518 29794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:36:23.490754 29794 hybrid_clock.cc:648] HybridClock initialized: now 1781980583490674 us; error 53 us; skew 500 ppm
I20260620 18:36:23.491057 29794 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:36:23 dist-test-slave-5lgp krb5kdc[28766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980583, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.37.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:36:23.494567 29794 init.cc:377] Logged in from keytab as oryx/127.22.37.2@KRBTEST.COM (short username oryx)
I20260620 18:36:23.496541 29794 webserver.cc:460] Webserver started at http://127.22.37.2:40713/ using document root <none> and password file <none>
I20260620 18:36:23.497185 29794 fs_manager.cc:359] Metadata directory not provided
I20260620 18:36:23.497354 29794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:36:23.502897 29794 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 18:36:23.505780 29809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:23.507144 29794 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 18:36:23.507280 29794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/ts-1/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/ts-1/wal
uuid: "519801f0627c402c9a65ccab106caf51"
format_stamp: "Formatted at 2026-06-20 18:36:20 on dist-test-slave-5lgp"
server_key: "184f67a051d793b3a6d90d320c0a173d"
server_key_iv: "f0ad1ae698cc5b84a59dd7be5030c73f"
server_key_version: "encryptionkey@0"
I20260620 18:36:23.508400 29794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:36:23.531090 29794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:36:23.535115 29794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:36:23.535385 29794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:36:23.536337 29794 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:36:23.537801 29794 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:36:23.537907 29794 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:23.537997 29794 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:36:23.538057 29794 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:23.572417 29794 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.2:33791
I20260620 18:36:23.572468 29921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.2:33791 every 8 connection(s)
I20260620 18:36:23.574059 29794 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/ts-1/data/info.pb
I20260620 18:36:23.574473 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 29794
I20260620 18:36:23.574766 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 29139
Jun 20 18:36:23 dist-test-slave-5lgp krb5kdc[28766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980583, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.37.2@KRBTEST.COM for oryx/127.22.37.61@KRBTEST.COM
Jun 20 18:36:23 dist-test-slave-5lgp krb5kdc[28766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980583, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.37.2@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM
Jun 20 18:36:23 dist-test-slave-5lgp krb5kdc[28766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980583, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.37.2@KRBTEST.COM for oryx/127.22.37.60@KRBTEST.COM
Jun 20 18:36:23 dist-test-slave-5lgp krb5kdc[28766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980578, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.37.3@KRBTEST.COM
I20260620 18:36:23.612013 29931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:23.578608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.2:39565 (local address 127.22.37.62:46099)
0620 18:36:23.588856 (+ 10248us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:23.588864 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:23.588889 (+ 25us) server_negotiation.cc:407] Connection header received
0620 18:36:23.589429 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:23.589436 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:23.589527 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:23.589637 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:23.592865 (+ 3228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:23.593957 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:23.596615 (+ 2658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:23.597267 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:23.597451 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:23.603355 (+ 5904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:23.603363 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:23.603369 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:23.603420 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:23.607487 (+ 4067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:23.608830 (+ 1343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:23.608837 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:23.608840 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:23.609079 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:23.609691 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:23.609696 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:23.609701 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:23.610085 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:23.610236 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:23.611332 (+ 1096us) server_negotiation.cc:299] Negotiation successful
0620 18:36:23.611558 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10117,"thread_start_us":10011,"threads_started":1}
I20260620 18:36:23.612514 29932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:23.578886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.2:45877 (local address 127.22.37.60:44179)
0620 18:36:23.579441 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:23.579452 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:23.582126 (+ 2674us) server_negotiation.cc:407] Connection header received
0620 18:36:23.589430 (+ 7304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:23.589436 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:23.589527 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:23.589620 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:23.592506 (+ 2886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:23.593753 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:23.597043 (+ 3290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:23.598108 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:23.598200 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:23.603865 (+ 5665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:23.603872 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:23.603876 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:23.603921 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:23.607751 (+ 3830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:23.608801 (+ 1050us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:23.608807 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:23.608810 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:23.608969 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:23.609721 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:23.609725 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:23.609728 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:23.610416 (+ 688us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:23.610527 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:23.611331 (+ 804us) server_negotiation.cc:299] Negotiation successful
0620 18:36:23.611783 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":246,"threads_started":1}
I20260620 18:36:23.613157 29929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:23.577948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:46099 (local address 127.22.37.2:39565)
0620 18:36:23.588090 (+ 10142us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:23.588126 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:23.588281 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:23.590428 (+ 2147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:23.590434 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:23.592282 (+ 1848us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:23.592686 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:23.592698 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:23.594159 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:23.594165 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:23.596414 (+ 2249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:23.596434 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:23.598476 (+ 2042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:23.598483 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:23.598980 (+ 497us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:23.599816 (+ 836us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:23.599838 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:23.603096 (+ 3258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:36:23.607671 (+ 4575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:23.607682 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:23.607699 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:23.608676 (+ 977us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:23.609217 (+ 541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:23.609220 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:23.609222 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:23.609485 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:23.610361 (+ 876us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:36:23.610369 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:36:23.610868 (+ 499us) client_negotiation.cc:769] Sending connection context
0620 18:36:23.612300 (+ 1432us) client_negotiation.cc:240] Negotiation successful
0620 18:36:23.612634 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9954,"mutex_wait_us":483,"spinlock_wait_cycles":14592,"thread_start_us":10915,"threads_started":1}
I20260620 18:36:23.613929 29930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:23.578218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.2:39157 (local address 127.22.37.61:44853)
0620 18:36:23.590833 (+ 12615us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:23.590839 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:23.590856 (+ 17us) server_negotiation.cc:407] Connection header received
0620 18:36:23.590926 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:23.590931 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:23.591007 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:23.591108 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:23.592537 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:23.593736 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:23.596395 (+ 2659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:23.597034 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:23.597150 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:23.602701 (+ 5551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:23.602707 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:23.602712 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:23.602753 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:23.607605 (+ 4852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:23.608801 (+ 1196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:23.608807 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:23.608810 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:23.609170 (+ 360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:23.609849 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:23.609854 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:23.609858 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:23.610229 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:23.610382 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:23.613243 (+ 2861us) server_negotiation.cc:299] Negotiation successful
0620 18:36:23.613498 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12480,"thread_start_us":12373,"threads_started":1}
I20260620 18:36:23.614180 29926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:23.576091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55884 (local address 127.22.37.3:35311)
0620 18:36:23.590092 (+ 14001us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:23.590105 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:23.590859 (+ 754us) server_negotiation.cc:407] Connection header received
0620 18:36:23.590931 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:23.590939 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:23.591223 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:23.591394 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:23.593246 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:23.594796 (+ 1550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:23.596856 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:23.597761 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:23.597930 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:23.604701 (+ 6771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:23.604716 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:23.604732 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:23.604781 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:23.608929 (+ 4148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:23.609862 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:23.609870 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:23.609873 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:23.610269 (+ 396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:23.610729 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:23.610734 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:23.610737 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:23.611123 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:23.613019 (+ 1896us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:23.613165 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 18:36:23.613580 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13792,"thread_start_us":13602,"threads_started":1}
I20260620 18:36:23.614265 29928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:23.577828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.61:44853 (local address 127.22.37.2:39157)
0620 18:36:23.589228 (+ 11400us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:23.589249 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:23.589407 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:23.591263 (+ 1856us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:23.591268 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:23.591841 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:23.592335 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:23.592355 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:23.593928 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:23.593933 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:23.596232 (+ 2299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:23.596250 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:23.597464 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:23.597472 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:23.598574 (+ 1102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:23.599354 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:23.599393 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:23.602494 (+ 3101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:36:23.607776 (+ 5282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:23.607783 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:23.607790 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:23.608676 (+ 886us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:23.609482 (+ 806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:23.609487 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:23.609491 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:23.609696 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:23.610438 (+ 742us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:36:23.610443 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:36:23.610869 (+ 426us) client_negotiation.cc:769] Sending connection context
0620 18:36:23.611211 (+ 342us) client_negotiation.cc:240] Negotiation successful
0620 18:36:23.611688 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11206,"mutex_wait_us":1736,"spinlock_wait_cycles":33792,"thread_start_us":1000,"threads_started":1}
I20260620 18:36:23.614651 29933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:23.577829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.60:44179 (local address 127.22.37.2:45877)
0620 18:36:23.580802 (+ 2973us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:23.580887 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:23.586559 (+ 5672us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:23.590305 (+ 3746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:23.590324 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:23.591263 (+ 939us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:23.592275 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:23.592300 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:23.593956 (+ 1656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:23.593962 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:23.596881 (+ 2919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:23.596892 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:23.598713 (+ 1821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:23.598719 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:23.598910 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:23.599435 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:23.599455 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:23.603660 (+ 4205us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:36:23.607873 (+ 4213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:23.607877 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:23.607880 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:23.608667 (+ 787us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:23.609312 (+ 645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:23.609317 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:23.609321 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:23.609586 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:23.610618 (+ 1032us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:36:23.610622 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:36:23.610872 (+ 250us) client_negotiation.cc:769] Sending connection context
0620 18:36:23.613574 (+ 2702us) client_negotiation.cc:240] Negotiation successful
0620 18:36:23.613848 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2581,"mutex_wait_us":1012,"spinlock_wait_cycles":109952,"thread_start_us":9704,"threads_started":1}
I20260620 18:36:23.614897 29923 heartbeater.cc:347] Connected to a master server at 127.22.37.62:46099
I20260620 18:36:23.615211 29923 heartbeater.cc:464] Registering TS with master...
I20260620 18:36:23.615260 29924 heartbeater.cc:347] Connected to a master server at 127.22.37.61:44853
I20260620 18:36:23.615411 29924 heartbeater.cc:464] Registering TS with master...
I20260620 18:36:23.615767 29262 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:36:23.616181 29923 heartbeater.cc:511] Master 127.22.37.62:46099 requested a full tablet report, sending...
I20260620 18:36:23.616218 29924 heartbeater.cc:511] Master 127.22.37.61:44853 requested a full tablet report, sending...
I20260620 18:36:23.617790 29922 heartbeater.cc:347] Connected to a master server at 127.22.37.60:44179
I20260620 18:36:23.617921 29922 heartbeater.cc:464] Registering TS with master...
I20260620 18:36:23.618155 28811 ts_manager.cc:194] Re-registered known tserver with Master: 519801f0627c402c9a65ccab106caf51 (127.22.37.2:33791)
I20260620 18:36:23.618306 29922 heartbeater.cc:511] Master 127.22.37.60:44179 requested a full tablet report, sending...
I20260620 18:36:23.619145 29416 ts_manager.cc:194] Registered new tserver with Master: 519801f0627c402c9a65ccab106caf51 (127.22.37.2:33791)
I20260620 18:36:23.619257 28811 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.37.2@KRBTEST.COM'} at 127.22.37.2:39565
I20260620 18:36:23.619310 29598 ts_manager.cc:194] Registered new tserver with Master: 519801f0627c402c9a65ccab106caf51 (127.22.37.2:33791)
I20260620 18:36:23.703631 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 29139
I20260620 18:36:23.719750 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.37.3:35311
--local_ip_for_outbound_sockets=127.22.37.3
--tserver_master_addrs=127.22.37.62:46099
--webserver_port=44253
--webserver_interface=127.22.37.3
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/krb5kdc/oryx_127.22.37.3.keytab
--principal=oryx/127.22.37.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:38175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.22.37.62:46099,127.22.37.61:44853
--tserver_master_addrs=127.22.37.62:46099,127.22.37.61:44853,127.22.37.60:44179 with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:36:24.010929 29778 heartbeater.cc:503] Master 127.22.37.62:46099 was elected leader, sending a full tablet report...
W20260620 18:36:24.046700 29935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:36:24.047117 29935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:36:24.047302 29935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:36:24.055104 29935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:36:24.055275 29935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:36:24.055500 29935 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.37.3
I20260620 18:36:24.065654 29935 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:38175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/krb5kdc/oryx_127.22.37.3.keytab
--principal=oryx/127.22.37.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.37.3:35311
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.37.3
--webserver_port=44253
--tserver_master_addrs=127.22.37.62:46099,127.22.37.61:44853,127.22.37.60:44179
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.37.3
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:36:24.067083 29935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:36:24.068537 29935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:36:24.082719 29941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:24.082659 29940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:24.083140 29943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:24.083457 29935 server_base.cc:1034] running on GCE node
I20260620 18:36:24.084255 29935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:36:24.085263 29935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:36:24.086545 29935 hybrid_clock.cc:648] HybridClock initialized: now 1781980584086512 us; error 71 us; skew 500 ppm
I20260620 18:36:24.086894 29935 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:36:24 dist-test-slave-5lgp krb5kdc[28766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980584, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.37.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:36:24.090248 29935 init.cc:377] Logged in from keytab as oryx/127.22.37.3@KRBTEST.COM (short username oryx)
I20260620 18:36:24.091711 29935 webserver.cc:460] Webserver started at http://127.22.37.3:44253/ using document root <none> and password file <none>
I20260620 18:36:24.092414 29935 fs_manager.cc:359] Metadata directory not provided
I20260620 18:36:24.092585 29935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:36:24.098089 29935 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:36:24.100713 29950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:24.102346 29935 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 18:36:24.102566 29935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/ts-2/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/ts-2/wal
uuid: "75525021a76043b8a1227ca817b38ec8"
format_stamp: "Formatted at 2026-06-20 18:36:20 on dist-test-slave-5lgp"
server_key: "71069479db998d2d45290018ed95bb3c"
server_key_iv: "14c7e1bfe4ce9098c85b0f762fd6f487"
server_key_version: "encryptionkey@0"
I20260620 18:36:24.103545 29935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:36:24.119262 29935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:36:24.123418 29935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:36:24.123719 29935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:36:24.124678 29935 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:36:24.126256 29935 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:36:24.126379 29935 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:24.126513 29935 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:36:24.126608 29935 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:24.160712 29935 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.3:35311
I20260620 18:36:24.160802 30062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.3:35311 every 8 connection(s)
I20260620 18:36:24.162256 29935 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/ts-2/data/info.pb
I20260620 18:36:24.170421 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 29935
I20260620 18:36:24.216302 30082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:24.176525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43878 (local address 127.22.37.62:46099)
0620 18:36:24.184738 (+ 8213us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:24.184747 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:24.184774 (+ 27us) server_negotiation.cc:407] Connection header received
0620 18:36:24.184854 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:24.184862 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:24.184991 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:24.185985 (+ 994us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:24.189298 (+ 3313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:24.192657 (+ 3359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:24.196549 (+ 3892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:24.198002 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:24.198173 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:24.207247 (+ 9074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:24.207258 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:24.207266 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:24.207324 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:24.212586 (+ 5262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:24.213561 (+ 975us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:24.213566 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:24.213571 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:24.213705 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:24.214172 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:24.214176 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:24.214178 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:24.214437 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:24.214681 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:24.215693 (+ 1012us) server_negotiation.cc:299] Negotiation successful
0620 18:36:24.215906 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8051,"thread_start_us":7929,"threads_started":1}
I20260620 18:36:24.222360 28811 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43878:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:36:24.234448 28811 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
Jun 20 18:36:24 dist-test-slave-5lgp krb5kdc[28766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980584, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.37.3@KRBTEST.COM for oryx/127.22.37.60@KRBTEST.COM
Jun 20 18:36:24 dist-test-slave-5lgp krb5kdc[28766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980584, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.37.3@KRBTEST.COM for oryx/127.22.37.61@KRBTEST.COM
I20260620 18:36:24.239279 30079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:24.175534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41186 (local address 127.22.37.61:44853)
0620 18:36:24.203263 (+ 27729us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:24.203273 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:24.203302 (+ 29us) server_negotiation.cc:407] Connection header received
0620 18:36:24.203396 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:24.203404 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:24.203511 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:24.203673 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:24.214946 (+ 11273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:24.216101 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:24.217822 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:24.218465 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:24.218573 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:24.227979 (+ 9406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:24.227986 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:24.227991 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:24.228030 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:24.233469 (+ 5439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:24.234315 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:24.234320 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:24.234324 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:24.234495 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:24.236970 (+ 2475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:24.236975 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:24.236978 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:24.237331 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:24.237631 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:24.238390 (+ 759us) server_negotiation.cc:299] Negotiation successful
0620 18:36:24.238697 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27538,"thread_start_us":28264,"threads_started":1}
Jun 20 18:36:24 dist-test-slave-5lgp krb5kdc[28766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980584, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.37.3@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM
I20260620 18:36:24.246757 30068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:24.165710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.3:60765 (local address 127.22.37.60:44179)
0620 18:36:24.166576 (+ 866us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:24.166584 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:24.218734 (+ 52150us) server_negotiation.cc:407] Connection header received
0620 18:36:24.219695 (+ 961us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:24.219701 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:24.219780 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:24.219860 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:24.223207 (+ 3347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:24.224424 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:24.228511 (+ 4087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:24.229123 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:24.229229 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:24.236132 (+ 6903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:24.236139 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:24.236144 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:24.236179 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:24.239742 (+ 3563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:24.242569 (+ 2827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:24.242582 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:24.242589 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:24.242739 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:24.244633 (+ 1894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:24.244640 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:24.244645 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:24.244986 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:24.245113 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:24.246164 (+ 1051us) server_negotiation.cc:299] Negotiation successful
0620 18:36:24.246406 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":731,"thread_start_us":649,"threads_started":1}
I20260620 18:36:24.247521 30080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:24.176055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40558 (local address 127.22.37.60:44179)
0620 18:36:24.206947 (+ 30892us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:24.206956 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:24.214987 (+ 8031us) server_negotiation.cc:407] Connection header received
0620 18:36:24.215268 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:24.215277 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:24.215400 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:24.215496 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:24.219901 (+ 4405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:24.221087 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:24.223092 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:24.223850 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:24.223941 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:24.233978 (+ 10037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:24.233987 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:24.233996 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:24.234039 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:24.240582 (+ 6543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:24.243980 (+ 3398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:24.243986 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:24.243992 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:24.244157 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:24.245148 (+ 991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:24.245152 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:24.245156 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:24.245495 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:24.245702 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:24.246298 (+ 596us) server_negotiation.cc:299] Negotiation successful
0620 18:36:24.247105 (+ 807us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":644,"server-negotiator.queue_time_us":30706,"thread_start_us":30917,"threads_started":1}
I20260620 18:36:24.247903 30071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:24.166697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.3:33457 (local address 127.22.37.61:44853)
0620 18:36:24.167142 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:24.167146 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:24.206465 (+ 39319us) server_negotiation.cc:407] Connection header received
0620 18:36:24.220088 (+ 13623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:24.220094 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:24.220173 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:24.220247 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:24.227978 (+ 7731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:24.229261 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:24.231169 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:24.231805 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:24.231917 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:24.239676 (+ 7759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:24.239685 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:24.239690 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:24.239723 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:24.243851 (+ 4128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:24.244639 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:24.244647 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:24.244651 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:24.244799 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:24.245559 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:24.245566 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:24.245570 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:24.245898 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:24.246032 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:24.246591 (+ 559us) server_negotiation.cc:299] Negotiation successful
0620 18:36:24.247225 (+ 634us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":253,"threads_started":1}
I20260620 18:36:24.249091 30070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:24.166038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.60:44179 (local address 127.22.37.3:60765)
0620 18:36:24.218632 (+ 52594us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:24.218651 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:24.219322 (+ 671us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:24.220822 (+ 1500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:24.220826 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:24.222741 (+ 1915us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:24.223051 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:24.223066 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:24.224760 (+ 1694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:24.224764 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:24.228294 (+ 3530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:24.228307 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:24.229618 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:24.229624 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:24.229995 (+ 371us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:24.230730 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:24.230764 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:24.235869 (+ 5105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:36:24.240488 (+ 4619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:24.240497 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:24.240517 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:24.241133 (+ 616us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:24.243562 (+ 2429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:24.243573 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:24.243578 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:24.243809 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:24.245234 (+ 1425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:36:24.245244 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:36:24.245711 (+ 467us) client_negotiation.cc:769] Sending connection context
0620 18:36:24.247064 (+ 1353us) client_negotiation.cc:240] Negotiation successful
0620 18:36:24.247635 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52436,"mutex_wait_us":1583,"thread_start_us":42813,"threads_started":1}
I20260620 18:36:24.249643 30072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:24.166038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.61:44853 (local address 127.22.37.3:33457)
0620 18:36:24.206234 (+ 40196us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:24.206308 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:24.219937 (+ 13629us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:24.220491 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:24.220496 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:24.227488 (+ 6992us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:24.227734 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:24.227745 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:24.229493 (+ 1748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:24.229497 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:24.231008 (+ 1511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:24.231022 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:24.232752 (+ 1730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:24.232761 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:24.233406 (+ 645us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:24.236148 (+ 2742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:24.236170 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:24.239434 (+ 3264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:36:24.244019 (+ 4585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:24.244024 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:24.244027 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:24.244503 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:24.245192 (+ 689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:24.245199 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:24.245205 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:24.245428 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:24.246117 (+ 689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:36:24.246125 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:36:24.246295 (+ 170us) client_negotiation.cc:769] Sending connection context
0620 18:36:24.247692 (+ 1397us) client_negotiation.cc:240] Negotiation successful
0620 18:36:24.247965 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39708,"thread_start_us":52640,"threads_started":1}
I20260620 18:36:24.250895 30065 heartbeater.cc:347] Connected to a master server at 127.22.37.61:44853
I20260620 18:36:24.251236 30065 heartbeater.cc:464] Registering TS with master...
I20260620 18:36:24.252156 30065 heartbeater.cc:511] Master 127.22.37.61:44853 requested a full tablet report, sending...
I20260620 18:36:24.253219 30067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:24.165445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.3:32831 (local address 127.22.37.62:46099)
0620 18:36:24.166218 (+ 773us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:24.166227 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:24.212041 (+ 45814us) server_negotiation.cc:407] Connection header received
0620 18:36:24.219707 (+ 7666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:24.219713 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:24.219792 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:24.219919 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:24.222720 (+ 2801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:24.224012 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:24.230633 (+ 6621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:24.231405 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:24.231520 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:24.244086 (+ 12566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:24.244099 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:24.244106 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:24.244145 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:24.248754 (+ 4609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:24.249929 (+ 1175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:24.249934 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:24.249937 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:24.250096 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:24.250507 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:24.250511 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:24.250514 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:24.250846 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:24.251002 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:24.252352 (+ 1350us) server_negotiation.cc:299] Negotiation successful
0620 18:36:24.252647 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":500,"threads_started":1}
I20260620 18:36:24.254432 30063 heartbeater.cc:347] Connected to a master server at 127.22.37.60:44179
I20260620 18:36:24.254575 30063 heartbeater.cc:464] Registering TS with master...
I20260620 18:36:24.254711 30069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:24.166283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:46099 (local address 127.22.37.3:32831)
0620 18:36:24.211910 (+ 45627us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:24.211940 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:24.219338 (+ 7398us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:24.220393 (+ 1055us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:24.220427 (+ 34us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:24.221626 (+ 1199us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:24.222527 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:24.222552 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:24.224576 (+ 2024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:24.224581 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:24.230472 (+ 5891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:24.230485 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:24.232230 (+ 1745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:24.232238 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:24.232438 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:24.235589 (+ 3151us) spinlock_profiling.cc:243] Waited 1.23 ms on lock 0x60c0000349f8. stack: 00007f9699da066e 00007f9699da0430 00007f9698e275ed 00000000008ea064 00000000008ea02d 00000000008e99fa 00007f9699d960fd 00007f9699d96977 00007f96a08ad69d 00007f96a07ac810 00007f96a07a9470 00007f96a07a433a 00007f96a082bdca 00007f96a0829f68 00007f96a085d82f 00007f96a085d58d
0620 18:36:24.235613 (+ 24us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:24.235636 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:24.243896 (+ 8260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:36:24.248865 (+ 4969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:24.248870 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:24.248883 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:24.249418 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:24.250229 (+ 811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:24.250234 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:24.250237 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:24.250382 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:24.251062 (+ 680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:36:24.251066 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:36:24.251218 (+ 152us) client_negotiation.cc:769] Sending connection context
0620 18:36:24.251425 (+ 207us) client_negotiation.cc:240] Negotiation successful
0620 18:36:24.251851 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45458,"mutex_wait_us":3060,"spinlock_wait_cycles":2829696,"thread_start_us":52002,"threads_started":1}
I20260620 18:36:24.255081 30063 heartbeater.cc:511] Master 127.22.37.60:44179 requested a full tablet report, sending...
I20260620 18:36:24.256210 30064 heartbeater.cc:347] Connected to a master server at 127.22.37.62:46099
I20260620 18:36:24.256350 30064 heartbeater.cc:464] Registering TS with master...
I20260620 18:36:24.256680 30064 heartbeater.cc:511] Master 127.22.37.62:46099 requested a full tablet report, sending...
I20260620 18:36:24.257903 28810 ts_manager.cc:194] Re-registered known tserver with Master: 75525021a76043b8a1227ca817b38ec8 (127.22.37.3:35311)
I20260620 18:36:24.259572 28810 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.37.3@KRBTEST.COM'} at 127.22.37.3:32831
I20260620 18:36:24.262094 29416 ts_manager.cc:194] Registered new tserver with Master: 75525021a76043b8a1227ca817b38ec8 (127.22.37.3:35311)
I20260620 18:36:24.263942 29598 ts_manager.cc:194] Registered new tserver with Master: 75525021a76043b8a1227ca817b38ec8 (127.22.37.3:35311)
I20260620 18:36:24.303015 30098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:24.277785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.2:33791 (local address 127.0.0.1:56078)
0620 18:36:24.294525 (+ 16740us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:24.294539 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:24.294687 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:24.295747 (+ 1060us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:24.295751 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:24.295774 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:36:24.296207 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:24.296224 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:24.298137 (+ 1913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:24.298141 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:24.300192 (+ 2051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:24.300205 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:24.302239 (+ 2034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:24.302245 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:24.302407 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:24.302439 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 18:36:24.302529 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 18:36:24.302669 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16634,"mutex_wait_us":1,"thread_start_us":18272,"threads_started":1}
I20260620 18:36:24.303074 30097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:24.278001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56078 (local address 127.22.37.2:33791)
0620 18:36:24.295022 (+ 17021us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:24.295029 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:24.295053 (+ 24us) server_negotiation.cc:407] Connection header received
0620 18:36:24.295168 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:24.295176 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:24.295531 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:24.295750 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:36:24.296437 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:24.297963 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:24.300384 (+ 2421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:24.301756 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:24.301872 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:24.302089 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:24.302615 (+ 526us) server_negotiation.cc:299] Negotiation successful
0620 18:36:24.302784 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16898,"thread_start_us":16835,"threads_started":1}
I20260620 18:36:24.305652 30096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:24.278334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.3:35311 (local address 127.0.0.1:55900)
0620 18:36:24.296384 (+ 18050us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:24.296407 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:24.296515 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:24.297735 (+ 1220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:24.297739 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:24.297765 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:36:24.298239 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:24.298250 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:24.300612 (+ 2362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:24.300617 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:24.302673 (+ 2056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:24.302683 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:24.304782 (+ 2099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:24.304788 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:24.304977 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:24.305009 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 18:36:24.305089 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 18:36:24.305216 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17908,"mutex_wait_us":2,"thread_start_us":15930,"threads_started":1}
I20260620 18:36:24.306294 30094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:24.277041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.1:35961 (local address 127.0.0.1:36620)
0620 18:36:24.292178 (+ 15137us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:24.292200 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:24.292353 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:24.297531 (+ 5178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:24.297535 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:24.297554 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:36:24.297884 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:24.297893 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:24.300059 (+ 2166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:24.300065 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:24.301747 (+ 1682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:24.301757 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:24.305507 (+ 3750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:24.305513 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:24.305716 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:24.305756 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 18:36:24.305850 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 18:36:24.305979 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14955,"thread_start_us":15424,"threads_started":1}
I20260620 18:36:24.306362 30095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:24.277250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36620 (local address 127.22.37.1:35961)
0620 18:36:24.296970 (+ 19720us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:24.296979 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:24.297004 (+ 25us) server_negotiation.cc:407] Connection header received
0620 18:36:24.297092 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:24.297101 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:24.297326 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:24.297539 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:36:24.298048 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:24.299894 (+ 1846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:24.301886 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:24.305259 (+ 3373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:24.305434 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:24.305635 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:24.305920 (+ 285us) server_negotiation.cc:299] Negotiation successful
0620 18:36:24.306081 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19558,"thread_start_us":19450,"threads_started":1}
I20260620 18:36:24.306790 30099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:24.278872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55900 (local address 127.22.37.3:35311)
0620 18:36:24.293978 (+ 15106us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:24.293991 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:24.296524 (+ 2533us) server_negotiation.cc:407] Connection header received
0620 18:36:24.296658 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:24.296670 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:24.296926 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:24.297089 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:36:24.298932 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:24.300482 (+ 1550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:24.302784 (+ 2302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:24.304588 (+ 1804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:24.304713 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:24.304919 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:24.305941 (+ 1022us) server_negotiation.cc:299] Negotiation successful
0620 18:36:24.306111 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14889,"thread_start_us":14704,"threads_started":1}
I20260620 18:36:24.307358 29857 tablet_service.cc:1467] Processing CreateTablet for tablet 34789350521c480ab9a1061239291461 (DEFAULT_TABLE table=test-table [id=e80aa4b379dd43da890bca36917105f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:36:24.308640 29857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34789350521c480ab9a1061239291461. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:36:24.309597 29712 tablet_service.cc:1467] Processing CreateTablet for tablet 34789350521c480ab9a1061239291461 (DEFAULT_TABLE table=test-table [id=e80aa4b379dd43da890bca36917105f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:36:24.311404 29712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34789350521c480ab9a1061239291461. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:36:24.320734 30100 tablet_bootstrap.cc:492] T 34789350521c480ab9a1061239291461 P 519801f0627c402c9a65ccab106caf51: Bootstrap starting.
I20260620 18:36:24.321403 29998 tablet_service.cc:1467] Processing CreateTablet for tablet 34789350521c480ab9a1061239291461 (DEFAULT_TABLE table=test-table [id=e80aa4b379dd43da890bca36917105f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:36:24.322775 29998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34789350521c480ab9a1061239291461. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:36:24.324517 30100 tablet_bootstrap.cc:654] T 34789350521c480ab9a1061239291461 P 519801f0627c402c9a65ccab106caf51: Neither blocks nor log segments found. Creating new log.
I20260620 18:36:24.324939 30101 tablet_bootstrap.cc:492] T 34789350521c480ab9a1061239291461 P 7232cd23c889418689f2c886ba008313: Bootstrap starting.
I20260620 18:36:24.325959 30100 log.cc:824] T 34789350521c480ab9a1061239291461 P 519801f0627c402c9a65ccab106caf51: Log is configured to *not* fsync() on all Append() calls
I20260620 18:36:24.329159 30101 tablet_bootstrap.cc:654] T 34789350521c480ab9a1061239291461 P 7232cd23c889418689f2c886ba008313: Neither blocks nor log segments found. Creating new log.
I20260620 18:36:24.330952 30101 log.cc:824] T 34789350521c480ab9a1061239291461 P 7232cd23c889418689f2c886ba008313: Log is configured to *not* fsync() on all Append() calls
I20260620 18:36:24.332649 30100 tablet_bootstrap.cc:492] T 34789350521c480ab9a1061239291461 P 519801f0627c402c9a65ccab106caf51: No bootstrap required, opened a new log
I20260620 18:36:24.332939 30100 ts_tablet_manager.cc:1397] T 34789350521c480ab9a1061239291461 P 519801f0627c402c9a65ccab106caf51: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.004s
I20260620 18:36:24.333961 30101 tablet_bootstrap.cc:492] T 34789350521c480ab9a1061239291461 P 7232cd23c889418689f2c886ba008313: No bootstrap required, opened a new log
I20260620 18:36:24.334197 30101 ts_tablet_manager.cc:1397] T 34789350521c480ab9a1061239291461 P 7232cd23c889418689f2c886ba008313: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.007s
I20260620 18:36:24.338268 30104 tablet_bootstrap.cc:492] T 34789350521c480ab9a1061239291461 P 75525021a76043b8a1227ca817b38ec8: Bootstrap starting.
I20260620 18:36:24.341208 30104 tablet_bootstrap.cc:654] T 34789350521c480ab9a1061239291461 P 75525021a76043b8a1227ca817b38ec8: Neither blocks nor log segments found. Creating new log.
I20260620 18:36:24.342223 30104 log.cc:824] T 34789350521c480ab9a1061239291461 P 75525021a76043b8a1227ca817b38ec8: Log is configured to *not* fsync() on all Append() calls
I20260620 18:36:24.343850 30100 raft_consensus.cc:348] T 34789350521c480ab9a1061239291461 P 519801f0627c402c9a65ccab106caf51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7232cd23c889418689f2c886ba008313" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 35961 } } peers { permanent_uuid: "519801f0627c402c9a65ccab106caf51" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 33791 } } peers { permanent_uuid: "75525021a76043b8a1227ca817b38ec8" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35311 } }
I20260620 18:36:24.344280 30100 raft_consensus.cc:374] T 34789350521c480ab9a1061239291461 P 519801f0627c402c9a65ccab106caf51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:36:24.344375 30100 raft_consensus.cc:729] T 34789350521c480ab9a1061239291461 P 519801f0627c402c9a65ccab106caf51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 519801f0627c402c9a65ccab106caf51, State: Initialized, Role: FOLLOWER
I20260620 18:36:24.344901 30104 tablet_bootstrap.cc:492] T 34789350521c480ab9a1061239291461 P 75525021a76043b8a1227ca817b38ec8: No bootstrap required, opened a new log
I20260620 18:36:24.345175 30104 ts_tablet_manager.cc:1397] T 34789350521c480ab9a1061239291461 P 75525021a76043b8a1227ca817b38ec8: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260620 18:36:24.344991 30100 consensus_queue.cc:260] T 34789350521c480ab9a1061239291461 P 519801f0627c402c9a65ccab106caf51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7232cd23c889418689f2c886ba008313" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 35961 } } peers { permanent_uuid: "519801f0627c402c9a65ccab106caf51" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 33791 } } peers { permanent_uuid: "75525021a76043b8a1227ca817b38ec8" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35311 } }
I20260620 18:36:24.345244 30101 raft_consensus.cc:348] T 34789350521c480ab9a1061239291461 P 7232cd23c889418689f2c886ba008313 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7232cd23c889418689f2c886ba008313" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 35961 } } peers { permanent_uuid: "519801f0627c402c9a65ccab106caf51" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 33791 } } peers { permanent_uuid: "75525021a76043b8a1227ca817b38ec8" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35311 } }
I20260620 18:36:24.345765 30101 raft_consensus.cc:374] T 34789350521c480ab9a1061239291461 P 7232cd23c889418689f2c886ba008313 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:36:24.345875 30101 raft_consensus.cc:729] T 34789350521c480ab9a1061239291461 P 7232cd23c889418689f2c886ba008313 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7232cd23c889418689f2c886ba008313, State: Initialized, Role: FOLLOWER
I20260620 18:36:24.346818 30101 consensus_queue.cc:260] T 34789350521c480ab9a1061239291461 P 7232cd23c889418689f2c886ba008313 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7232cd23c889418689f2c886ba008313" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 35961 } } peers { permanent_uuid: "519801f0627c402c9a65ccab106caf51" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 33791 } } peers { permanent_uuid: "75525021a76043b8a1227ca817b38ec8" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35311 } }
I20260620 18:36:24.353538 30100 ts_tablet_manager.cc:1428] T 34789350521c480ab9a1061239291461 P 519801f0627c402c9a65ccab106caf51: Time spent starting tablet: real 0.020s user 0.015s sys 0.005s
I20260620 18:36:24.356423 30104 raft_consensus.cc:348] T 34789350521c480ab9a1061239291461 P 75525021a76043b8a1227ca817b38ec8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7232cd23c889418689f2c886ba008313" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 35961 } } peers { permanent_uuid: "519801f0627c402c9a65ccab106caf51" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 33791 } } peers { permanent_uuid: "75525021a76043b8a1227ca817b38ec8" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35311 } }
I20260620 18:36:24.356854 30104 raft_consensus.cc:374] T 34789350521c480ab9a1061239291461 P 75525021a76043b8a1227ca817b38ec8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:36:24.356964 30104 raft_consensus.cc:729] T 34789350521c480ab9a1061239291461 P 75525021a76043b8a1227ca817b38ec8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75525021a76043b8a1227ca817b38ec8, State: Initialized, Role: FOLLOWER
I20260620 18:36:24.357697 30104 consensus_queue.cc:260] T 34789350521c480ab9a1061239291461 P 75525021a76043b8a1227ca817b38ec8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7232cd23c889418689f2c886ba008313" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 35961 } } peers { permanent_uuid: "519801f0627c402c9a65ccab106caf51" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 33791 } } peers { permanent_uuid: "75525021a76043b8a1227ca817b38ec8" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35311 } }
I20260620 18:36:24.360178 29923 heartbeater.cc:503] Master 127.22.37.62:46099 was elected leader, sending a full tablet report...
I20260620 18:36:24.368009 30104 ts_tablet_manager.cc:1428] T 34789350521c480ab9a1061239291461 P 75525021a76043b8a1227ca817b38ec8: Time spent starting tablet: real 0.023s user 0.016s sys 0.004s
I20260620 18:36:24.368467 30064 heartbeater.cc:503] Master 127.22.37.62:46099 was elected leader, sending a full tablet report...
I20260620 18:36:24.368603 30101 ts_tablet_manager.cc:1428] T 34789350521c480ab9a1061239291461 P 7232cd23c889418689f2c886ba008313: Time spent starting tablet: real 0.034s user 0.019s sys 0.014s
W20260620 18:36:24.414759 30066 tablet.cc:2369] T 34789350521c480ab9a1061239291461 P 75525021a76043b8a1227ca817b38ec8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:36:24.415449 30106 raft_consensus.cc:482] T 34789350521c480ab9a1061239291461 P 519801f0627c402c9a65ccab106caf51 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:36:24.415755 30106 raft_consensus.cc:504] T 34789350521c480ab9a1061239291461 P 519801f0627c402c9a65ccab106caf51 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7232cd23c889418689f2c886ba008313" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 35961 } } peers { permanent_uuid: "519801f0627c402c9a65ccab106caf51" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 33791 } } peers { permanent_uuid: "75525021a76043b8a1227ca817b38ec8" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35311 } }
I20260620 18:36:24.417354 30106 leader_election.cc:288] T 34789350521c480ab9a1061239291461 P 519801f0627c402c9a65ccab106caf51 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7232cd23c889418689f2c886ba008313 (127.22.37.1:35961), 75525021a76043b8a1227ca817b38ec8 (127.22.37.3:35311)
I20260620 18:36:24.424341 30095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:24.417652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.2:54257 (local address 127.22.37.1:35961)
0620 18:36:24.417902 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:24.417908 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:24.418345 (+ 437us) server_negotiation.cc:407] Connection header received
0620 18:36:24.418543 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:24.418549 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:24.418626 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:24.418714 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:36:24.419311 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:24.420470 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:24.422616 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:24.423365 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:24.423470 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:24.423601 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:24.423939 (+ 338us) server_negotiation.cc:299] Negotiation successful
0620 18:36:24.424087 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 18:36:24.424373 30099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:24.418035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.2:45251 (local address 127.22.37.3:35311)
0620 18:36:24.418251 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:24.418257 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:24.418575 (+ 318us) server_negotiation.cc:407] Connection header received
0620 18:36:24.418751 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:24.418756 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:24.418833 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:24.418935 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:36:24.419915 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:24.420634 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:24.422650 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:24.423445 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:24.423578 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:24.423678 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:24.424012 (+ 334us) server_negotiation.cc:299] Negotiation successful
0620 18:36:24.424157 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 18:36:24.424427 30109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:24.417546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.1:35961 (local address 127.22.37.2:54257)
0620 18:36:24.418228 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:24.418251 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:24.418402 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:24.418746 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:24.418751 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:24.418777 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:36:24.419167 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:24.419178 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:24.420617 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:24.420622 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:24.422459 (+ 1837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:24.422471 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:24.423581 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:24.423596 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:24.423772 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:24.423816 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 18:36:24.423887 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 18:36:24.424036 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"mutex_wait_us":28,"thread_start_us":318,"threads_started":1}
I20260620 18:36:24.424718 30110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:24.417624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.3:35311 (local address 127.22.37.2:45251)
0620 18:36:24.418437 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:24.418470 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:24.418602 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:24.418957 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:24.418960 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:24.418979 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:36:24.419295 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:24.419305 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:24.420773 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:24.420777 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:24.422459 (+ 1682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:24.422472 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:24.423670 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:24.423675 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:24.423785 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:24.423857 (+ 72us) client_negotiation.cc:769] Sending connection context
0620 18:36:24.423966 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 18:36:24.424086 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":697,"mutex_wait_us":60,"thread_start_us":535,"threads_started":1}
I20260620 18:36:24.425540 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34789350521c480ab9a1061239291461" candidate_uuid: "519801f0627c402c9a65ccab106caf51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7232cd23c889418689f2c886ba008313" is_pre_election: true
I20260620 18:36:24.426064 29732 raft_consensus.cc:2454] T 34789350521c480ab9a1061239291461 P 7232cd23c889418689f2c886ba008313 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 519801f0627c402c9a65ccab106caf51 in term 0.
I20260620 18:36:24.425956 30018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34789350521c480ab9a1061239291461" candidate_uuid: "519801f0627c402c9a65ccab106caf51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75525021a76043b8a1227ca817b38ec8" is_pre_election: true
I20260620 18:36:24.426504 30018 raft_consensus.cc:2454] T 34789350521c480ab9a1061239291461 P 75525021a76043b8a1227ca817b38ec8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 519801f0627c402c9a65ccab106caf51 in term 0.
I20260620 18:36:24.426781 29812 leader_election.cc:302] T 34789350521c480ab9a1061239291461 P 519801f0627c402c9a65ccab106caf51 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 519801f0627c402c9a65ccab106caf51, 7232cd23c889418689f2c886ba008313; no voters:
I20260620 18:36:24.427233 30106 raft_consensus.cc:2781] T 34789350521c480ab9a1061239291461 P 519801f0627c402c9a65ccab106caf51 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:36:24.427388 30106 raft_consensus.cc:482] T 34789350521c480ab9a1061239291461 P 519801f0627c402c9a65ccab106caf51 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:36:24.427495 30106 raft_consensus.cc:3037] T 34789350521c480ab9a1061239291461 P 519801f0627c402c9a65ccab106caf51 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:36:24.429895 30106 raft_consensus.cc:504] T 34789350521c480ab9a1061239291461 P 519801f0627c402c9a65ccab106caf51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7232cd23c889418689f2c886ba008313" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 35961 } } peers { permanent_uuid: "519801f0627c402c9a65ccab106caf51" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 33791 } } peers { permanent_uuid: "75525021a76043b8a1227ca817b38ec8" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35311 } }
I20260620 18:36:24.430476 30106 leader_election.cc:288] T 34789350521c480ab9a1061239291461 P 519801f0627c402c9a65ccab106caf51 [CANDIDATE]: Term 1 election: Requested vote from peers 7232cd23c889418689f2c886ba008313 (127.22.37.1:35961), 75525021a76043b8a1227ca817b38ec8 (127.22.37.3:35311)
I20260620 18:36:24.431000 30018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34789350521c480ab9a1061239291461" candidate_uuid: "519801f0627c402c9a65ccab106caf51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75525021a76043b8a1227ca817b38ec8"
I20260620 18:36:24.430991 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34789350521c480ab9a1061239291461" candidate_uuid: "519801f0627c402c9a65ccab106caf51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7232cd23c889418689f2c886ba008313"
I20260620 18:36:24.431201 29732 raft_consensus.cc:3037] T 34789350521c480ab9a1061239291461 P 7232cd23c889418689f2c886ba008313 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:36:24.431221 30018 raft_consensus.cc:3037] T 34789350521c480ab9a1061239291461 P 75525021a76043b8a1227ca817b38ec8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:36:24.434845 30018 raft_consensus.cc:2454] T 34789350521c480ab9a1061239291461 P 75525021a76043b8a1227ca817b38ec8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 519801f0627c402c9a65ccab106caf51 in term 1.
I20260620 18:36:24.434845 29732 raft_consensus.cc:2454] T 34789350521c480ab9a1061239291461 P 7232cd23c889418689f2c886ba008313 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 519801f0627c402c9a65ccab106caf51 in term 1.
I20260620 18:36:24.435281 29812 leader_election.cc:302] T 34789350521c480ab9a1061239291461 P 519801f0627c402c9a65ccab106caf51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 519801f0627c402c9a65ccab106caf51, 7232cd23c889418689f2c886ba008313; no voters:
I20260620 18:36:24.435580 30106 raft_consensus.cc:2781] T 34789350521c480ab9a1061239291461 P 519801f0627c402c9a65ccab106caf51 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:36:24.436035 30106 raft_consensus.cc:686] T 34789350521c480ab9a1061239291461 P 519801f0627c402c9a65ccab106caf51 [term 1 LEADER]: Becoming Leader. State: Replica: 519801f0627c402c9a65ccab106caf51, State: Running, Role: LEADER
I20260620 18:36:24.436573 30106 consensus_queue.cc:237] T 34789350521c480ab9a1061239291461 P 519801f0627c402c9a65ccab106caf51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7232cd23c889418689f2c886ba008313" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 35961 } } peers { permanent_uuid: "519801f0627c402c9a65ccab106caf51" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 33791 } } peers { permanent_uuid: "75525021a76043b8a1227ca817b38ec8" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35311 } }
I20260620 18:36:24.441666 28811 catalog_manager.cc:5495] T 34789350521c480ab9a1061239291461 P 519801f0627c402c9a65ccab106caf51 reported cstate change: term changed from 0 to 1, leader changed from <none> to 519801f0627c402c9a65ccab106caf51 (127.22.37.2). New cstate: current_term: 1 leader_uuid: "519801f0627c402c9a65ccab106caf51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7232cd23c889418689f2c886ba008313" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 35961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "519801f0627c402c9a65ccab106caf51" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 33791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "75525021a76043b8a1227ca817b38ec8" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35311 } health_report { overall_health: UNKNOWN } } }
W20260620 18:36:24.476332 29780 tablet.cc:2369] T 34789350521c480ab9a1061239291461 P 7232cd23c889418689f2c886ba008313: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:36:24.490274 30097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:24.484488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56086 (local address 127.22.37.2:33791)
0620 18:36:24.484697 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:24.484703 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:24.484723 (+ 20us) server_negotiation.cc:407] Connection header received
0620 18:36:24.484934 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:24.484938 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:24.485026 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:24.485185 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:36:24.485612 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:24.486339 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:24.488061 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:24.488472 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:24.488605 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:24.489133 (+ 528us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:36:24.489719 (+ 586us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:36:24.489915 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:24.490008 (+ 93us) server_negotiation.cc:299] Negotiation successful
0620 18:36:24.490076 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 18:36:24.498641 29732 raft_consensus.cc:1261] T 34789350521c480ab9a1061239291461 P 7232cd23c889418689f2c886ba008313 [term 1 FOLLOWER]: Refusing update from remote peer 519801f0627c402c9a65ccab106caf51: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:36:24.498641 30018 raft_consensus.cc:1261] T 34789350521c480ab9a1061239291461 P 75525021a76043b8a1227ca817b38ec8 [term 1 FOLLOWER]: Refusing update from remote peer 519801f0627c402c9a65ccab106caf51: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:36:24.499641 30111 consensus_queue.cc:1035] T 34789350521c480ab9a1061239291461 P 519801f0627c402c9a65ccab106caf51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75525021a76043b8a1227ca817b38ec8" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:36:24.500212 30106 consensus_queue.cc:1035] T 34789350521c480ab9a1061239291461 P 519801f0627c402c9a65ccab106caf51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7232cd23c889418689f2c886ba008313" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 35961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:36:24.518985 30114 mvcc.cc:205] Tried to move back new op lower bound from 7298992474095964160 to 7298992473856352256. Current Snapshot: MvccSnapshot[applied={T|T < 7298992474095964160}]
I20260620 18:36:24.519622 30116 mvcc.cc:205] Tried to move back new op lower bound from 7298992474095964160 to 7298992473856352256. Current Snapshot: MvccSnapshot[applied={T|T < 7298992474095964160}]
I20260620 18:36:24.521004 30115 mvcc.cc:205] Tried to move back new op lower bound from 7298992474095964160 to 7298992473856352256. Current Snapshot: MvccSnapshot[applied={T|T < 7298992474095964160}]
I20260620 18:36:24.545801 28811 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43878:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:36:24.546094 28811 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43878:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:36:24.552675 28811 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c8fe34680f3c4d388068736f074cb839: Sending DeleteTablet for 3 replicas of tablet 34789350521c480ab9a1061239291461
I20260620 18:36:24.553910 29712 tablet_service.cc:1514] Processing DeleteTablet for tablet 34789350521c480ab9a1061239291461 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:36:24 UTC) from {username='oryx', principal='oryx/127.22.37.62@KRBTEST.COM'} at 127.0.0.1:36620
I20260620 18:36:24.554519 29857 tablet_service.cc:1514] Processing DeleteTablet for tablet 34789350521c480ab9a1061239291461 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:36:24 UTC) from {username='oryx', principal='oryx/127.22.37.62@KRBTEST.COM'} at 127.0.0.1:56078
I20260620 18:36:24.555151 29998 tablet_service.cc:1514] Processing DeleteTablet for tablet 34789350521c480ab9a1061239291461 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:36:24 UTC) from {username='oryx', principal='oryx/127.22.37.62@KRBTEST.COM'} at 127.0.0.1:55900
I20260620 18:36:24.555559 30124 tablet_replica.cc:330] T 34789350521c480ab9a1061239291461 P 7232cd23c889418689f2c886ba008313: stopping tablet replica
I20260620 18:36:24.556022 30124 raft_consensus.cc:2229] T 34789350521c480ab9a1061239291461 P 7232cd23c889418689f2c886ba008313 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:36:24.556648 30124 raft_consensus.cc:2258] T 34789350521c480ab9a1061239291461 P 7232cd23c889418689f2c886ba008313 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:36:24.559091 30124 ts_tablet_manager.cc:1905] T 34789350521c480ab9a1061239291461 P 7232cd23c889418689f2c886ba008313: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:36:24.568943 30124 ts_tablet_manager.cc:1918] T 34789350521c480ab9a1061239291461 P 7232cd23c889418689f2c886ba008313: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:36:24.569149 30124 log.cc:1196] T 34789350521c480ab9a1061239291461 P 7232cd23c889418689f2c886ba008313: Deleting WAL directory at /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/ts-0/wal/wals/34789350521c480ab9a1061239291461
I20260620 18:36:24.569821 30124 ts_tablet_manager.cc:1939] T 34789350521c480ab9a1061239291461 P 7232cd23c889418689f2c886ba008313: Deleting consensus metadata
I20260620 18:36:24.571332 30125 tablet_replica.cc:330] T 34789350521c480ab9a1061239291461 P 519801f0627c402c9a65ccab106caf51: stopping tablet replica
I20260620 18:36:24.571941 30125 raft_consensus.cc:2229] T 34789350521c480ab9a1061239291461 P 519801f0627c402c9a65ccab106caf51 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:36:24.572194 28797 catalog_manager.cc:4841] TS 7232cd23c889418689f2c886ba008313 (127.22.37.1:35961): tablet 34789350521c480ab9a1061239291461 (table test-table [id=e80aa4b379dd43da890bca36917105f5]) successfully deleted
I20260620 18:36:24.572716 30125 raft_consensus.cc:2258] T 34789350521c480ab9a1061239291461 P 519801f0627c402c9a65ccab106caf51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:36:24.575003 30125 ts_tablet_manager.cc:1905] T 34789350521c480ab9a1061239291461 P 519801f0627c402c9a65ccab106caf51: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:36:24.583715 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 28779
I20260620 18:36:24.585717 28841 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:36:24.585912 30125 ts_tablet_manager.cc:1918] T 34789350521c480ab9a1061239291461 P 519801f0627c402c9a65ccab106caf51: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:36:24.586131 30125 log.cc:1196] T 34789350521c480ab9a1061239291461 P 519801f0627c402c9a65ccab106caf51: Deleting WAL directory at /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/ts-1/wal/wals/34789350521c480ab9a1061239291461
I20260620 18:36:24.587078 30126 tablet_replica.cc:330] T 34789350521c480ab9a1061239291461 P 75525021a76043b8a1227ca817b38ec8: stopping tablet replica
I20260620 18:36:24.588117 30125 ts_tablet_manager.cc:1939] T 34789350521c480ab9a1061239291461 P 519801f0627c402c9a65ccab106caf51: Deleting consensus metadata
I20260620 18:36:24.589110 30126 raft_consensus.cc:2229] T 34789350521c480ab9a1061239291461 P 75525021a76043b8a1227ca817b38ec8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:36:24.590416 30126 raft_consensus.cc:2258] T 34789350521c480ab9a1061239291461 P 75525021a76043b8a1227ca817b38ec8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:36:24.596946 30126 ts_tablet_manager.cc:1905] T 34789350521c480ab9a1061239291461 P 75525021a76043b8a1227ca817b38ec8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:36:24.613970 30126 ts_tablet_manager.cc:1918] T 34789350521c480ab9a1061239291461 P 75525021a76043b8a1227ca817b38ec8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:36:24.614210 30126 log.cc:1196] T 34789350521c480ab9a1061239291461 P 75525021a76043b8a1227ca817b38ec8: Deleting WAL directory at /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/ts-2/wal/wals/34789350521c480ab9a1061239291461
I20260620 18:36:24.614786 30126 ts_tablet_manager.cc:1939] T 34789350521c480ab9a1061239291461 P 75525021a76043b8a1227ca817b38ec8: Deleting consensus metadata
I20260620 18:36:24.615162 30071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:24.557925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41196 (local address 127.22.37.61:44853)
0620 18:36:24.558110 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:24.558117 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:24.582023 (+ 23906us) server_negotiation.cc:407] Connection header received
0620 18:36:24.582259 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:24.582264 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:24.582362 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:24.582459 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:24.586381 (+ 3922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:24.587525 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:24.589929 (+ 2404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:24.590578 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:24.590701 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:24.606734 (+ 16033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:24.606745 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:24.606753 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:24.606814 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:24.611313 (+ 4499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:24.612422 (+ 1109us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:24.612429 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:24.612433 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:24.612622 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:24.613208 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:24.613214 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:24.613217 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:24.613588 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:24.613918 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:24.614540 (+ 622us) server_negotiation.cc:299] Negotiation successful
0620 18:36:24.614794 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 18:36:24.630502 30080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:24.582819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40574 (local address 127.22.37.60:44179)
0620 18:36:24.583029 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:24.583035 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:24.616852 (+ 33817us) server_negotiation.cc:407] Connection header received
0620 18:36:24.617096 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:24.617103 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:24.617187 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:24.617241 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:24.618993 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:24.620050 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:24.621507 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:24.621940 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:24.622094 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:24.624599 (+ 2505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:24.624603 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:24.624607 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:24.624638 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:24.627613 (+ 2975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:24.628350 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:24.628356 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:24.628359 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:24.628492 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:24.628912 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:24.628917 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:24.628920 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:24.629186 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:24.629474 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:24.629865 (+ 391us) server_negotiation.cc:299] Negotiation successful
0620 18:36:24.630082 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 18:36:24.790005 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 28779
I20260620 18:36:24.852478 30071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:24.827792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41210 (local address 127.22.37.61:44853)
0620 18:36:24.828009 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:24.828016 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:24.836377 (+ 8361us) server_negotiation.cc:407] Connection header received
0620 18:36:24.836592 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:24.836597 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:24.836674 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:24.836779 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:24.839416 (+ 2637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:24.840583 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:24.842092 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:24.842511 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:24.842648 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:24.845765 (+ 3117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:24.845770 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:24.845773 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:24.845802 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:24.849233 (+ 3431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:24.850135 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:24.850140 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:24.850144 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:24.850281 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:24.850678 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:24.850682 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:24.850686 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:24.851074 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:24.851300 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:24.851878 (+ 578us) server_negotiation.cc:299] Negotiation successful
0620 18:36:24.852123 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 18:36:24.853413 30080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:24.834629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40576 (local address 127.22.37.60:44179)
0620 18:36:24.834815 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:24.834824 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:24.837212 (+ 2388us) server_negotiation.cc:407] Connection header received
0620 18:36:24.837489 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:24.837495 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:24.837593 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:24.837669 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:24.839890 (+ 2221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:24.840748 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:24.842695 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:24.843297 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:24.843397 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:24.846883 (+ 3486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:24.846889 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:24.846893 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:24.846933 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:24.850124 (+ 3191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:24.850896 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:24.850901 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:24.850904 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:24.851044 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:24.851436 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:24.851441 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:24.851444 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:24.851789 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:24.852008 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:24.852812 (+ 804us) server_negotiation.cc:299] Negotiation successful
0620 18:36:24.853135 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 18:36:25.531709 30136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:25.530982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:46099 (local address 127.22.37.2:47707)
0620 18:36:25.531431 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:25.531550 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:46099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":216,"threads_started":1}
W20260620 18:36:25.532125 29923 heartbeater.cc:644] Failed to heartbeat to 127.22.37.62:46099 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.22.37.62:46099: connect: Connection refused (error 111)
I20260620 18:36:25.536052 30137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:25.535222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:46099 (local address 127.22.37.1:54573)
0620 18:36:25.535794 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:25.535909 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:46099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":335,"threads_started":1}
W20260620 18:36:25.536628 29778 heartbeater.cc:644] Failed to heartbeat to 127.22.37.62:46099 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.22.37.62:46099: connect: Connection refused (error 111)
I20260620 18:36:25.538272 30138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:25.536946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:46099 (local address 127.22.37.3:32959)
0620 18:36:25.537855 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:25.538005 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:46099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":789,"thread_start_us":647,"threads_started":1}
W20260620 18:36:25.538645 30064 heartbeater.cc:644] Failed to heartbeat to 127.22.37.62:46099 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.22.37.62:46099: connect: Connection refused (error 111)
I20260620 18:36:26.053975 30139 raft_consensus.cc:482] T 00000000000000000000000000000000 P b1f845ed42fb4dcd90063c45dedac043 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c8fe34680f3c4d388068736f074cb839)
I20260620 18:36:26.054319 30140 raft_consensus.cc:482] T 00000000000000000000000000000000 P e3b608bf35ad4fa2ab340274d7920a6c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c8fe34680f3c4d388068736f074cb839)
I20260620 18:36:26.054296 30139 raft_consensus.cc:504] T 00000000000000000000000000000000 P b1f845ed42fb4dcd90063c45dedac043 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c8fe34680f3c4d388068736f074cb839" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 46099 } } peers { permanent_uuid: "b1f845ed42fb4dcd90063c45dedac043" member_type: VOTER last_known_addr { host: "127.22.37.61" port: 44853 } attrs { promote: false } } peers { permanent_uuid: "e3b608bf35ad4fa2ab340274d7920a6c" member_type: VOTER last_known_addr { host: "127.22.37.60" port: 44179 } attrs { promote: false } }
I20260620 18:36:26.054566 30140 raft_consensus.cc:504] T 00000000000000000000000000000000 P e3b608bf35ad4fa2ab340274d7920a6c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c8fe34680f3c4d388068736f074cb839" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 46099 } } peers { permanent_uuid: "b1f845ed42fb4dcd90063c45dedac043" member_type: VOTER last_known_addr { host: "127.22.37.61" port: 44853 } attrs { promote: false } } peers { permanent_uuid: "e3b608bf35ad4fa2ab340274d7920a6c" member_type: VOTER last_known_addr { host: "127.22.37.60" port: 44179 } attrs { promote: false } }
I20260620 18:36:26.055265 30139 leader_election.cc:288] T 00000000000000000000000000000000 P b1f845ed42fb4dcd90063c45dedac043 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c8fe34680f3c4d388068736f074cb839 (127.22.37.62:46099), e3b608bf35ad4fa2ab340274d7920a6c (127.22.37.60:44179)
I20260620 18:36:26.056339 30140 leader_election.cc:288] T 00000000000000000000000000000000 P e3b608bf35ad4fa2ab340274d7920a6c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c8fe34680f3c4d388068736f074cb839 (127.22.37.62:46099), b1f845ed42fb4dcd90063c45dedac043 (127.22.37.61:44853)
I20260620 18:36:26.057565 30142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:26.055461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:46099 (local address 127.0.0.1:44368)
0620 18:36:26.056982 (+ 1521us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:26.057192 (+ 210us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:46099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1359,"threads_started":1}
I20260620 18:36:26.067905 30142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:26.055721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.60:44179 (local address 127.0.0.1:40592)
0620 18:36:26.057808 (+ 2087us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:26.057831 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:26.058002 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:26.059209 (+ 1207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:26.059217 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:26.059255 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:36:26.059526 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:26.059541 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:26.061547 (+ 2006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:26.061556 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:26.064432 (+ 2876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:26.064455 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:26.066966 (+ 2511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:26.066975 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:26.067226 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:26.067272 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 18:36:26.067380 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 18:36:26.067534 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1966,"thread_start_us":6456,"threads_started":1}
I20260620 18:36:26.067946 30145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:26.056388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:46099 (local address 127.0.0.1:44380)
0620 18:36:26.067518 (+ 11130us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:26.067695 (+ 177us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:46099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":10982,"threads_started":1}
I20260620 18:36:26.068717 30143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:26.055710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40592 (local address 127.22.37.60:44179)
0620 18:36:26.058604 (+ 2894us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:26.058613 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:26.058647 (+ 34us) server_negotiation.cc:407] Connection header received
0620 18:36:26.058733 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:26.058741 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:26.058845 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:26.058960 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:36:26.059757 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:26.061321 (+ 1564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:26.064741 (+ 3420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:26.066487 (+ 1746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:26.066654 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:26.066828 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:26.068119 (+ 1291us) server_negotiation.cc:299] Negotiation successful
0620 18:36:26.068258 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2750,"thread_start_us":2562,"threads_started":1}
I20260620 18:36:26.069470 29608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b1f845ed42fb4dcd90063c45dedac043" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e3b608bf35ad4fa2ab340274d7920a6c" is_pre_election: true
W20260620 18:36:26.069585 29582 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.37.62:46099: connect: Connection refused (error 111)
W20260620 18:36:26.069833 29400 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.37.62:46099: connect: Connection refused (error 111)
I20260620 18:36:26.069921 29608 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e3b608bf35ad4fa2ab340274d7920a6c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b1f845ed42fb4dcd90063c45dedac043 in term 1.
I20260620 18:36:26.070474 29402 leader_election.cc:302] T 00000000000000000000000000000000 P b1f845ed42fb4dcd90063c45dedac043 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b1f845ed42fb4dcd90063c45dedac043, e3b608bf35ad4fa2ab340274d7920a6c; no voters:
I20260620 18:36:26.071225 30139 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b1f845ed42fb4dcd90063c45dedac043 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 18:36:26.071357 30139 raft_consensus.cc:482] T 00000000000000000000000000000000 P b1f845ed42fb4dcd90063c45dedac043 [term 1 FOLLOWER]: Starting leader election (detected failure of leader c8fe34680f3c4d388068736f074cb839)
I20260620 18:36:26.071484 30139 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b1f845ed42fb4dcd90063c45dedac043 [term 1 FOLLOWER]: Advancing to term 2
I20260620 18:36:26.072471 30144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:26.071501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:46099 (local address 127.0.0.1:44390)
0620 18:36:26.072112 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:26.072279 (+ 167us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:46099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":498}
I20260620 18:36:26.072773 30141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:26.072169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:46099 (local address 127.0.0.1:44404)
0620 18:36:26.072337 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:26.072529 (+ 192us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:46099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260620 18:36:26.072748 29582 leader_election.cc:334] T 00000000000000000000000000000000 P e3b608bf35ad4fa2ab340274d7920a6c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c8fe34680f3c4d388068736f074cb839 (127.22.37.62:46099): Network error: Client connection negotiation failed: client connection to 127.22.37.62:46099: connect: Connection refused (error 111)
W20260620 18:36:26.074659 29400 leader_election.cc:334] T 00000000000000000000000000000000 P b1f845ed42fb4dcd90063c45dedac043 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c8fe34680f3c4d388068736f074cb839 (127.22.37.62:46099): Network error: Client connection negotiation failed: client connection to 127.22.37.62:46099: connect: Connection refused (error 111)
I20260620 18:36:26.074595 30139 raft_consensus.cc:504] T 00000000000000000000000000000000 P b1f845ed42fb4dcd90063c45dedac043 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c8fe34680f3c4d388068736f074cb839" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 46099 } } peers { permanent_uuid: "b1f845ed42fb4dcd90063c45dedac043" member_type: VOTER last_known_addr { host: "127.22.37.61" port: 44853 } attrs { promote: false } } peers { permanent_uuid: "e3b608bf35ad4fa2ab340274d7920a6c" member_type: VOTER last_known_addr { host: "127.22.37.60" port: 44179 } attrs { promote: false } }
I20260620 18:36:26.075407 30139 leader_election.cc:288] T 00000000000000000000000000000000 P b1f845ed42fb4dcd90063c45dedac043 [CANDIDATE]: Term 2 election: Requested vote from peers c8fe34680f3c4d388068736f074cb839 (127.22.37.62:46099), e3b608bf35ad4fa2ab340274d7920a6c (127.22.37.60:44179)
I20260620 18:36:26.075871 30141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:26.075489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:46099 (local address 127.0.0.1:44432)
0620 18:36:26.075608 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:26.075704 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:46099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260620 18:36:26.075966 29608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b1f845ed42fb4dcd90063c45dedac043" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e3b608bf35ad4fa2ab340274d7920a6c"
I20260620 18:36:26.076164 29608 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e3b608bf35ad4fa2ab340274d7920a6c [term 1 FOLLOWER]: Advancing to term 2
I20260620 18:36:26.076922 30141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:26.076575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:46099 (local address 127.0.0.1:44448)
0620 18:36:26.076700 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:26.076772 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:46099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260620 18:36:26.077210 29400 leader_election.cc:334] T 00000000000000000000000000000000 P b1f845ed42fb4dcd90063c45dedac043 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer c8fe34680f3c4d388068736f074cb839 (127.22.37.62:46099): Network error: Client connection negotiation failed: client connection to 127.22.37.62:46099: connect: Connection refused (error 111)
I20260620 18:36:26.077504 30146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:26.062576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41212 (local address 127.22.37.61:44853)
0620 18:36:26.067951 (+ 5375us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:26.067960 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:26.068291 (+ 331us) server_negotiation.cc:407] Connection header received
0620 18:36:26.068372 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:26.068379 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:26.068479 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:26.068600 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:36:26.070541 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:26.072074 (+ 1533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:26.075125 (+ 3051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:26.076485 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:26.076621 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:26.076737 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:26.077009 (+ 272us) server_negotiation.cc:299] Negotiation successful
0620 18:36:26.077139 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5224,"thread_start_us":4015,"threads_started":1}
I20260620 18:36:26.077585 30145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:26.058616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.61:44853 (local address 127.0.0.1:41212)
0620 18:36:26.068108 (+ 9492us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:26.068126 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:26.068249 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:26.069919 (+ 1670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:26.069924 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:26.069956 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:36:26.070339 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:26.070350 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:26.072960 (+ 2610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:26.072965 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:26.074864 (+ 1899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:26.074877 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:26.076673 (+ 1796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:26.076679 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:26.076851 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:26.076874 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 18:36:26.077025 (+ 151us) client_negotiation.cc:240] Negotiation successful
0620 18:36:26.077178 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9401,"thread_start_us":8617,"threads_started":1}
I20260620 18:36:26.078186 29426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e3b608bf35ad4fa2ab340274d7920a6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "b1f845ed42fb4dcd90063c45dedac043" is_pre_election: true
I20260620 18:36:26.078482 29426 raft_consensus.cc:2379] T 00000000000000000000000000000000 P b1f845ed42fb4dcd90063c45dedac043 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3b608bf35ad4fa2ab340274d7920a6c in current term 2: Already voted for candidate b1f845ed42fb4dcd90063c45dedac043 in this term.
I20260620 18:36:26.078958 29608 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e3b608bf35ad4fa2ab340274d7920a6c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1f845ed42fb4dcd90063c45dedac043 in term 2.
I20260620 18:36:26.079062 29583 leader_election.cc:302] T 00000000000000000000000000000000 P e3b608bf35ad4fa2ab340274d7920a6c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e3b608bf35ad4fa2ab340274d7920a6c; no voters: b1f845ed42fb4dcd90063c45dedac043, c8fe34680f3c4d388068736f074cb839
I20260620 18:36:26.079442 30140 raft_consensus.cc:2726] T 00000000000000000000000000000000 P e3b608bf35ad4fa2ab340274d7920a6c [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 18:36:26.079800 29402 leader_election.cc:302] T 00000000000000000000000000000000 P b1f845ed42fb4dcd90063c45dedac043 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b1f845ed42fb4dcd90063c45dedac043, e3b608bf35ad4fa2ab340274d7920a6c; no voters: c8fe34680f3c4d388068736f074cb839
I20260620 18:36:26.080127 30139 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b1f845ed42fb4dcd90063c45dedac043 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 18:36:26.080550 30139 raft_consensus.cc:686] T 00000000000000000000000000000000 P b1f845ed42fb4dcd90063c45dedac043 [term 2 LEADER]: Becoming Leader. State: Replica: b1f845ed42fb4dcd90063c45dedac043, State: Running, Role: LEADER
I20260620 18:36:26.080930 30139 consensus_queue.cc:237] T 00000000000000000000000000000000 P b1f845ed42fb4dcd90063c45dedac043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c8fe34680f3c4d388068736f074cb839" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 46099 } } peers { permanent_uuid: "b1f845ed42fb4dcd90063c45dedac043" member_type: VOTER last_known_addr { host: "127.22.37.61" port: 44853 } attrs { promote: false } } peers { permanent_uuid: "e3b608bf35ad4fa2ab340274d7920a6c" member_type: VOTER last_known_addr { host: "127.22.37.60" port: 44179 } attrs { promote: false } }
I20260620 18:36:26.082571 30149 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1f845ed42fb4dcd90063c45dedac043 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b1f845ed42fb4dcd90063c45dedac043. Latest consensus state: current_term: 2 leader_uuid: "b1f845ed42fb4dcd90063c45dedac043" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c8fe34680f3c4d388068736f074cb839" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 46099 } } peers { permanent_uuid: "b1f845ed42fb4dcd90063c45dedac043" member_type: VOTER last_known_addr { host: "127.22.37.61" port: 44853 } attrs { promote: false } } peers { permanent_uuid: "e3b608bf35ad4fa2ab340274d7920a6c" member_type: VOTER last_known_addr { host: "127.22.37.60" port: 44179 } attrs { promote: false } } }
I20260620 18:36:26.082980 30149 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1f845ed42fb4dcd90063c45dedac043 [sys.catalog]: This master's current role is: LEADER
I20260620 18:36:26.083634 30151 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:36:26.087354 30151 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:36:26.088156 30151 catalog_manager.cc:1259] Loaded cluster ID: eee720b350864e718f512e0ff17d1a2e
I20260620 18:36:26.088254 30151 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:36:26.089422 30151 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:36:26.090210 30151 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b1f845ed42fb4dcd90063c45dedac043: Loaded TSK: 0
I20260620 18:36:26.091042 30151 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:36:26.132404 29416 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41210:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:36:26.139495 29416 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:36:26.143769 30141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:26.143342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:46099 (local address 127.0.0.1:44464)
0620 18:36:26.143523 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:26.143627 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:46099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260620 18:36:26.143900 29608 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e3b608bf35ad4fa2ab340274d7920a6c [term 2 FOLLOWER]: Refusing update from remote peer b1f845ed42fb4dcd90063c45dedac043: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260620 18:36:26.144551 30149 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b1f845ed42fb4dcd90063c45dedac043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3b608bf35ad4fa2ab340274d7920a6c" member_type: VOTER last_known_addr { host: "127.22.37.60" port: 44179 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260620 18:36:26.144951 30141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:26.144559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:46099 (local address 127.0.0.1:44476)
0620 18:36:26.144714 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:26.144802 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:46099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 18:36:26.145126 29400 consensus_peers.cc:487] T 00000000000000000000000000000000 P b1f845ed42fb4dcd90063c45dedac043 -> Peer c8fe34680f3c4d388068736f074cb839 (127.22.37.62:46099): Couldn't send request to peer c8fe34680f3c4d388068736f074cb839. Status: Network error: Client connection negotiation failed: client connection to 127.22.37.62:46099: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 18:36:26.147342 30140 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3b608bf35ad4fa2ab340274d7920a6c [sys.catalog]: SysCatalogTable state changed. Reason: New leader b1f845ed42fb4dcd90063c45dedac043. Latest consensus state: current_term: 2 leader_uuid: "b1f845ed42fb4dcd90063c45dedac043" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c8fe34680f3c4d388068736f074cb839" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 46099 } } peers { permanent_uuid: "b1f845ed42fb4dcd90063c45dedac043" member_type: VOTER last_known_addr { host: "127.22.37.61" port: 44853 } attrs { promote: false } } peers { permanent_uuid: "e3b608bf35ad4fa2ab340274d7920a6c" member_type: VOTER last_known_addr { host: "127.22.37.60" port: 44179 } attrs { promote: false } } }
I20260620 18:36:26.147650 30140 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3b608bf35ad4fa2ab340274d7920a6c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:36:26.148978 30139 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1f845ed42fb4dcd90063c45dedac043 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "b1f845ed42fb4dcd90063c45dedac043" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c8fe34680f3c4d388068736f074cb839" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 46099 } } peers { permanent_uuid: "b1f845ed42fb4dcd90063c45dedac043" member_type: VOTER last_known_addr { host: "127.22.37.61" port: 44853 } attrs { promote: false } } peers { permanent_uuid: "e3b608bf35ad4fa2ab340274d7920a6c" member_type: VOTER last_known_addr { host: "127.22.37.60" port: 44179 } attrs { promote: false } } }
I20260620 18:36:26.149617 30139 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1f845ed42fb4dcd90063c45dedac043 [sys.catalog]: This master's current role is: LEADER
I20260620 18:36:26.150705 30140 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3b608bf35ad4fa2ab340274d7920a6c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "b1f845ed42fb4dcd90063c45dedac043" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c8fe34680f3c4d388068736f074cb839" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 46099 } } peers { permanent_uuid: "b1f845ed42fb4dcd90063c45dedac043" member_type: VOTER last_known_addr { host: "127.22.37.61" port: 44853 } attrs { promote: false } } peers { permanent_uuid: "e3b608bf35ad4fa2ab340274d7920a6c" member_type: VOTER last_known_addr { host: "127.22.37.60" port: 44179 } attrs { promote: false } } }
I20260620 18:36:26.150964 30140 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3b608bf35ad4fa2ab340274d7920a6c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:36:26.151165 30153 mvcc.cc:205] Tried to move back new op lower bound from 7298992480837361664 to 7298992480590938112. Current Snapshot: MvccSnapshot[applied={T|T < 7298992480837361664 or (T in {7298992480837361664})}]
I20260620 18:36:26.177397 30142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:26.158121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.3:35311 (local address 127.0.0.1:55904)
0620 18:36:26.158245 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:26.158261 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:26.158370 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:26.171164 (+ 12794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:26.171172 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:26.171210 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:36:26.171466 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:26.171479 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:26.173111 (+ 1632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:26.173116 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:26.175059 (+ 1943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:26.175070 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:26.176533 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:26.176537 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:26.176798 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:26.176823 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 18:36:26.176895 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 18:36:26.177068 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":34}
I20260620 18:36:26.177397 30161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:26.159160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56094 (local address 127.22.37.2:33791)
0620 18:36:26.169777 (+ 10617us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:26.169787 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:26.170337 (+ 550us) server_negotiation.cc:407] Connection header received
0620 18:36:26.170531 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:26.170538 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:26.170631 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:26.171347 (+ 716us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:36:26.171524 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:26.172538 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:26.174992 (+ 2454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:26.176052 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:26.176170 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:26.176303 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:26.176941 (+ 638us) server_negotiation.cc:299] Negotiation successful
0620 18:36:26.177082 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10463,"thread_start_us":10371,"threads_started":1}
I20260620 18:36:26.177402 30159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:26.158331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55904 (local address 127.22.37.3:35311)
0620 18:36:26.170800 (+ 12469us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:26.170805 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:26.170822 (+ 17us) server_negotiation.cc:407] Connection header received
0620 18:36:26.170888 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:26.170895 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:26.170987 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:26.171091 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:36:26.171683 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:26.172964 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:26.175208 (+ 2244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:26.176386 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:26.176519 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:26.176635 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:26.177002 (+ 367us) server_negotiation.cc:299] Negotiation successful
0620 18:36:26.177127 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12354,"thread_start_us":12244,"threads_started":1}
I20260620 18:36:26.178566 30160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:26.158771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.2:33791 (local address 127.0.0.1:56094)
0620 18:36:26.170230 (+ 11459us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:26.170244 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:26.170382 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:26.170775 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:26.170780 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:26.170805 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:36:26.171178 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:26.171191 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:26.172679 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:26.172684 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:26.174799 (+ 2115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:26.174812 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:26.176485 (+ 1673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:26.176492 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:26.176705 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:26.176803 (+ 98us) client_negotiation.cc:769] Sending connection context
0620 18:36:26.176896 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 18:36:26.178191 (+ 1295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11341,"mutex_wait_us":68,"thread_start_us":11132,"threads_started":1}
I20260620 18:36:26.178730 29998 tablet_service.cc:1467] Processing CreateTablet for tablet 199986f855a849ebb076e6dde8df6f83 (DEFAULT_TABLE table=test-table [id=6e36f8d4312d471dada837d9ba8b47bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:36:26.178871 30141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:26.157643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.1:35961 (local address 127.0.0.1:36634)
0620 18:36:26.157795 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:26.157815 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:26.157957 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:26.173279 (+ 15322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:26.173284 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:26.173307 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:36:26.173787 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:26.173796 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:26.174738 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:26.174744 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:26.176656 (+ 1912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:26.176674 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:26.178144 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:26.178149 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:26.178300 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:26.178320 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 18:36:26.178423 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 18:36:26.178544 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":121}
I20260620 18:36:26.178915 30158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:26.157754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36634 (local address 127.22.37.1:35961)
0620 18:36:26.172903 (+ 15149us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:26.172909 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:26.172928 (+ 19us) server_negotiation.cc:407] Connection header received
0620 18:36:26.173007 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:26.173015 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:26.173095 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:26.173212 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:36:26.173933 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:26.174606 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:26.176812 (+ 2206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:26.177909 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:26.178065 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:26.178177 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:26.178452 (+ 275us) server_negotiation.cc:299] Negotiation successful
0620 18:36:26.178648 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15033,"thread_start_us":15568,"threads_started":1}
I20260620 18:36:26.179188 29998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 199986f855a849ebb076e6dde8df6f83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:36:26.179713 29857 tablet_service.cc:1467] Processing CreateTablet for tablet 199986f855a849ebb076e6dde8df6f83 (DEFAULT_TABLE table=test-table [id=6e36f8d4312d471dada837d9ba8b47bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:36:26.180086 29857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 199986f855a849ebb076e6dde8df6f83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:36:26.184062 30162 tablet_bootstrap.cc:492] T 199986f855a849ebb076e6dde8df6f83 P 75525021a76043b8a1227ca817b38ec8: Bootstrap starting.
I20260620 18:36:26.184310 29712 tablet_service.cc:1467] Processing CreateTablet for tablet 199986f855a849ebb076e6dde8df6f83 (DEFAULT_TABLE table=test-table [id=6e36f8d4312d471dada837d9ba8b47bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:36:26.184736 29712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 199986f855a849ebb076e6dde8df6f83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:36:26.185163 30163 tablet_bootstrap.cc:492] T 199986f855a849ebb076e6dde8df6f83 P 519801f0627c402c9a65ccab106caf51: Bootstrap starting.
I20260620 18:36:26.185886 30162 tablet_bootstrap.cc:654] T 199986f855a849ebb076e6dde8df6f83 P 75525021a76043b8a1227ca817b38ec8: Neither blocks nor log segments found. Creating new log.
I20260620 18:36:26.186949 30163 tablet_bootstrap.cc:654] T 199986f855a849ebb076e6dde8df6f83 P 519801f0627c402c9a65ccab106caf51: Neither blocks nor log segments found. Creating new log.
I20260620 18:36:26.188015 30162 tablet_bootstrap.cc:492] T 199986f855a849ebb076e6dde8df6f83 P 75525021a76043b8a1227ca817b38ec8: No bootstrap required, opened a new log
I20260620 18:36:26.188163 30162 ts_tablet_manager.cc:1397] T 199986f855a849ebb076e6dde8df6f83 P 75525021a76043b8a1227ca817b38ec8: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260620 18:36:26.188973 30163 tablet_bootstrap.cc:492] T 199986f855a849ebb076e6dde8df6f83 P 519801f0627c402c9a65ccab106caf51: No bootstrap required, opened a new log
I20260620 18:36:26.189123 30163 ts_tablet_manager.cc:1397] T 199986f855a849ebb076e6dde8df6f83 P 519801f0627c402c9a65ccab106caf51: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260620 18:36:26.188988 30162 raft_consensus.cc:348] T 199986f855a849ebb076e6dde8df6f83 P 75525021a76043b8a1227ca817b38ec8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7232cd23c889418689f2c886ba008313" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 35961 } } peers { permanent_uuid: "75525021a76043b8a1227ca817b38ec8" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35311 } } peers { permanent_uuid: "519801f0627c402c9a65ccab106caf51" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 33791 } }
I20260620 18:36:26.189218 30162 raft_consensus.cc:374] T 199986f855a849ebb076e6dde8df6f83 P 75525021a76043b8a1227ca817b38ec8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:36:26.189319 30162 raft_consensus.cc:729] T 199986f855a849ebb076e6dde8df6f83 P 75525021a76043b8a1227ca817b38ec8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75525021a76043b8a1227ca817b38ec8, State: Initialized, Role: FOLLOWER
I20260620 18:36:26.189929 30163 raft_consensus.cc:348] T 199986f855a849ebb076e6dde8df6f83 P 519801f0627c402c9a65ccab106caf51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7232cd23c889418689f2c886ba008313" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 35961 } } peers { permanent_uuid: "75525021a76043b8a1227ca817b38ec8" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35311 } } peers { permanent_uuid: "519801f0627c402c9a65ccab106caf51" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 33791 } }
I20260620 18:36:26.190205 30163 raft_consensus.cc:374] T 199986f855a849ebb076e6dde8df6f83 P 519801f0627c402c9a65ccab106caf51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:36:26.190315 30163 raft_consensus.cc:729] T 199986f855a849ebb076e6dde8df6f83 P 519801f0627c402c9a65ccab106caf51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 519801f0627c402c9a65ccab106caf51, State: Initialized, Role: FOLLOWER
I20260620 18:36:26.190464 30162 consensus_queue.cc:260] T 199986f855a849ebb076e6dde8df6f83 P 75525021a76043b8a1227ca817b38ec8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7232cd23c889418689f2c886ba008313" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 35961 } } peers { permanent_uuid: "75525021a76043b8a1227ca817b38ec8" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35311 } } peers { permanent_uuid: "519801f0627c402c9a65ccab106caf51" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 33791 } }
I20260620 18:36:26.190663 30166 tablet_bootstrap.cc:492] T 199986f855a849ebb076e6dde8df6f83 P 7232cd23c889418689f2c886ba008313: Bootstrap starting.
I20260620 18:36:26.190863 30163 consensus_queue.cc:260] T 199986f855a849ebb076e6dde8df6f83 P 519801f0627c402c9a65ccab106caf51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7232cd23c889418689f2c886ba008313" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 35961 } } peers { permanent_uuid: "75525021a76043b8a1227ca817b38ec8" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35311 } } peers { permanent_uuid: "519801f0627c402c9a65ccab106caf51" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 33791 } }
I20260620 18:36:26.191308 30162 ts_tablet_manager.cc:1428] T 199986f855a849ebb076e6dde8df6f83 P 75525021a76043b8a1227ca817b38ec8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 18:36:26.191870 30163 ts_tablet_manager.cc:1428] T 199986f855a849ebb076e6dde8df6f83 P 519801f0627c402c9a65ccab106caf51: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 18:36:26.192472 30166 tablet_bootstrap.cc:654] T 199986f855a849ebb076e6dde8df6f83 P 7232cd23c889418689f2c886ba008313: Neither blocks nor log segments found. Creating new log.
I20260620 18:36:26.198936 30167 raft_consensus.cc:482] T 199986f855a849ebb076e6dde8df6f83 P 75525021a76043b8a1227ca817b38ec8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:36:26.199535 30065 heartbeater.cc:503] Master 127.22.37.61:44853 was elected leader, sending a full tablet report...
I20260620 18:36:26.207752 29924 heartbeater.cc:503] Master 127.22.37.61:44853 was elected leader, sending a full tablet report...
I20260620 18:36:26.199535 30167 raft_consensus.cc:504] T 199986f855a849ebb076e6dde8df6f83 P 75525021a76043b8a1227ca817b38ec8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7232cd23c889418689f2c886ba008313" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 35961 } } peers { permanent_uuid: "75525021a76043b8a1227ca817b38ec8" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35311 } } peers { permanent_uuid: "519801f0627c402c9a65ccab106caf51" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 33791 } }
I20260620 18:36:26.210147 30169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:26.192349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:46099 (local address 127.22.37.2:46245)
0620 18:36:26.207231 (+ 14882us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:26.207436 (+ 205us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:46099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14691,"threads_started":1}
I20260620 18:36:26.213209 30169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:26.211950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:46099 (local address 127.22.37.2:42067)
0620 18:36:26.212643 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:26.212718 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:46099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":605,"threads_started":1}
I20260620 18:36:26.222549 30174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:26.192365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:46099 (local address 127.22.37.3:47891)
0620 18:36:26.221920 (+ 29555us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:26.222140 (+ 220us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:46099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29380,"threads_started":1}
I20260620 18:36:26.223140 30167 leader_election.cc:288] T 199986f855a849ebb076e6dde8df6f83 P 75525021a76043b8a1227ca817b38ec8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7232cd23c889418689f2c886ba008313 (127.22.37.1:35961), 519801f0627c402c9a65ccab106caf51 (127.22.37.2:33791)
I20260620 18:36:26.224812 30170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:26.224306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:46099 (local address 127.22.37.3:51211)
0620 18:36:26.224485 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:26.224615 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:46099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260620 18:36:26.226356 30166 tablet_bootstrap.cc:492] T 199986f855a849ebb076e6dde8df6f83 P 7232cd23c889418689f2c886ba008313: No bootstrap required, opened a new log
I20260620 18:36:26.226526 30166 ts_tablet_manager.cc:1397] T 199986f855a849ebb076e6dde8df6f83 P 7232cd23c889418689f2c886ba008313: Time spent bootstrapping tablet: real 0.036s user 0.012s sys 0.021s
I20260620 18:36:26.227535 30166 raft_consensus.cc:348] T 199986f855a849ebb076e6dde8df6f83 P 7232cd23c889418689f2c886ba008313 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7232cd23c889418689f2c886ba008313" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 35961 } } peers { permanent_uuid: "75525021a76043b8a1227ca817b38ec8" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35311 } } peers { permanent_uuid: "519801f0627c402c9a65ccab106caf51" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 33791 } }
I20260620 18:36:26.227887 30166 raft_consensus.cc:374] T 199986f855a849ebb076e6dde8df6f83 P 7232cd23c889418689f2c886ba008313 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:36:26.227998 30166 raft_consensus.cc:729] T 199986f855a849ebb076e6dde8df6f83 P 7232cd23c889418689f2c886ba008313 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7232cd23c889418689f2c886ba008313, State: Initialized, Role: FOLLOWER
I20260620 18:36:26.228430 30166 consensus_queue.cc:260] T 199986f855a849ebb076e6dde8df6f83 P 7232cd23c889418689f2c886ba008313 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7232cd23c889418689f2c886ba008313" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 35961 } } peers { permanent_uuid: "75525021a76043b8a1227ca817b38ec8" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35311 } } peers { permanent_uuid: "519801f0627c402c9a65ccab106caf51" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 33791 } }
I20260620 18:36:26.231144 30161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:26.212631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.3:49919 (local address 127.22.37.2:33791)
0620 18:36:26.212780 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:26.212787 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:26.222889 (+ 10102us) server_negotiation.cc:407] Connection header received
0620 18:36:26.223089 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:26.223094 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:26.223172 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:26.223254 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:36:26.223990 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:26.225010 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:26.228271 (+ 3261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:26.229668 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:26.229782 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:26.229907 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:26.230748 (+ 841us) server_negotiation.cc:299] Negotiation successful
0620 18:36:26.230885 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 18:36:26.231158 30174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:26.212241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.2:33791 (local address 127.22.37.3:49919)
0620 18:36:26.222783 (+ 10542us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:26.222805 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:26.222954 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:26.223462 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:26.223469 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:26.223510 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:36:26.223818 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:26.223830 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:26.225234 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:26.225242 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:26.228028 (+ 2786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:26.228049 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:26.230349 (+ 2300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:26.230355 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:26.230537 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:26.230572 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 18:36:26.230712 (+ 140us) client_negotiation.cc:240] Negotiation successful
0620 18:36:26.230858 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10427,"thread_start_us":9362,"threads_started":1}
I20260620 18:36:26.232018 29877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "199986f855a849ebb076e6dde8df6f83" candidate_uuid: "75525021a76043b8a1227ca817b38ec8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "519801f0627c402c9a65ccab106caf51" is_pre_election: true
I20260620 18:36:26.232426 29877 raft_consensus.cc:2454] T 199986f855a849ebb076e6dde8df6f83 P 519801f0627c402c9a65ccab106caf51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75525021a76043b8a1227ca817b38ec8 in term 0.
I20260620 18:36:26.232803 30166 ts_tablet_manager.cc:1428] T 199986f855a849ebb076e6dde8df6f83 P 7232cd23c889418689f2c886ba008313: Time spent starting tablet: real 0.006s user 0.001s sys 0.004s
I20260620 18:36:26.233021 29954 leader_election.cc:302] T 199986f855a849ebb076e6dde8df6f83 P 75525021a76043b8a1227ca817b38ec8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 519801f0627c402c9a65ccab106caf51, 75525021a76043b8a1227ca817b38ec8; no voters:
I20260620 18:36:26.233487 30172 raft_consensus.cc:2781] T 199986f855a849ebb076e6dde8df6f83 P 75525021a76043b8a1227ca817b38ec8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:36:26.233611 30172 raft_consensus.cc:482] T 199986f855a849ebb076e6dde8df6f83 P 75525021a76043b8a1227ca817b38ec8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:36:26.233727 30172 raft_consensus.cc:3037] T 199986f855a849ebb076e6dde8df6f83 P 75525021a76043b8a1227ca817b38ec8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:36:26.234633 30158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:26.224922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.3:57993 (local address 127.22.37.1:35961)
0620 18:36:26.225132 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:26.225138 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:26.225287 (+ 149us) server_negotiation.cc:407] Connection header received
0620 18:36:26.225606 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:26.225616 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:26.225689 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:26.225774 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:36:26.226582 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:26.227647 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:26.232110 (+ 4463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:26.233439 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:26.233600 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:26.233694 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:26.234347 (+ 653us) server_negotiation.cc:299] Negotiation successful
0620 18:36:26.234462 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 18:36:26.235262 30170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:26.224820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.1:35961 (local address 127.22.37.3:57993)
0620 18:36:26.225204 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:26.225223 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:26.225340 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:26.225978 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:26.225983 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:26.226006 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:36:26.226329 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:26.226343 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:26.228083 (+ 1740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:26.228088 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:26.231892 (+ 3804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:26.231912 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:26.233927 (+ 2015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:26.233934 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:26.234163 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:26.234200 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 18:36:26.234795 (+ 595us) client_negotiation.cc:240] Negotiation successful
0620 18:36:26.234917 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":4220,"threads_started":1}
I20260620 18:36:26.236367 29779 heartbeater.cc:503] Master 127.22.37.61:44853 was elected leader, sending a full tablet report...
I20260620 18:36:26.236449 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "199986f855a849ebb076e6dde8df6f83" candidate_uuid: "75525021a76043b8a1227ca817b38ec8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7232cd23c889418689f2c886ba008313" is_pre_election: true
I20260620 18:36:26.236557 30177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:26.234014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:46099 (local address 127.22.37.1:32903)
0620 18:36:26.236064 (+ 2050us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:26.236242 (+ 178us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:46099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1776,"threads_started":1}
I20260620 18:36:26.236692 29732 raft_consensus.cc:2454] T 199986f855a849ebb076e6dde8df6f83 P 7232cd23c889418689f2c886ba008313 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75525021a76043b8a1227ca817b38ec8 in term 0.
I20260620 18:36:26.237167 30172 raft_consensus.cc:504] T 199986f855a849ebb076e6dde8df6f83 P 75525021a76043b8a1227ca817b38ec8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7232cd23c889418689f2c886ba008313" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 35961 } } peers { permanent_uuid: "75525021a76043b8a1227ca817b38ec8" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35311 } } peers { permanent_uuid: "519801f0627c402c9a65ccab106caf51" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 33791 } }
I20260620 18:36:26.237928 30172 leader_election.cc:288] T 199986f855a849ebb076e6dde8df6f83 P 75525021a76043b8a1227ca817b38ec8 [CANDIDATE]: Term 1 election: Requested vote from peers 7232cd23c889418689f2c886ba008313 (127.22.37.1:35961), 519801f0627c402c9a65ccab106caf51 (127.22.37.2:33791)
I20260620 18:36:26.238546 29877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "199986f855a849ebb076e6dde8df6f83" candidate_uuid: "75525021a76043b8a1227ca817b38ec8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "519801f0627c402c9a65ccab106caf51"
I20260620 18:36:26.238754 29877 raft_consensus.cc:3037] T 199986f855a849ebb076e6dde8df6f83 P 519801f0627c402c9a65ccab106caf51 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:36:26.238873 30177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:26.238525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:46099 (local address 127.22.37.1:37355)
0620 18:36:26.238705 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:26.238768 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:46099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260620 18:36:26.238804 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "199986f855a849ebb076e6dde8df6f83" candidate_uuid: "75525021a76043b8a1227ca817b38ec8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7232cd23c889418689f2c886ba008313"
I20260620 18:36:26.239004 29732 raft_consensus.cc:3037] T 199986f855a849ebb076e6dde8df6f83 P 7232cd23c889418689f2c886ba008313 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:36:26.241778 29732 raft_consensus.cc:2454] T 199986f855a849ebb076e6dde8df6f83 P 7232cd23c889418689f2c886ba008313 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75525021a76043b8a1227ca817b38ec8 in term 1.
I20260620 18:36:26.241778 29877 raft_consensus.cc:2454] T 199986f855a849ebb076e6dde8df6f83 P 519801f0627c402c9a65ccab106caf51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75525021a76043b8a1227ca817b38ec8 in term 1.
I20260620 18:36:26.242139 29954 leader_election.cc:302] T 199986f855a849ebb076e6dde8df6f83 P 75525021a76043b8a1227ca817b38ec8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 519801f0627c402c9a65ccab106caf51, 75525021a76043b8a1227ca817b38ec8; no voters:
I20260620 18:36:26.242391 30172 raft_consensus.cc:2781] T 199986f855a849ebb076e6dde8df6f83 P 75525021a76043b8a1227ca817b38ec8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:36:26.242586 30172 raft_consensus.cc:686] T 199986f855a849ebb076e6dde8df6f83 P 75525021a76043b8a1227ca817b38ec8 [term 1 LEADER]: Becoming Leader. State: Replica: 75525021a76043b8a1227ca817b38ec8, State: Running, Role: LEADER
I20260620 18:36:26.242995 30172 consensus_queue.cc:237] T 199986f855a849ebb076e6dde8df6f83 P 75525021a76043b8a1227ca817b38ec8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7232cd23c889418689f2c886ba008313" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 35961 } } peers { permanent_uuid: "75525021a76043b8a1227ca817b38ec8" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35311 } } peers { permanent_uuid: "519801f0627c402c9a65ccab106caf51" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 33791 } }
I20260620 18:36:26.243389 30170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:26.242976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:46099 (local address 127.22.37.3:41513)
0620 18:36:26.243128 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:26.243244 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:46099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 18:36:26.246503 29416 catalog_manager.cc:5495] T 199986f855a849ebb076e6dde8df6f83 P 75525021a76043b8a1227ca817b38ec8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 75525021a76043b8a1227ca817b38ec8 (127.22.37.3). New cstate: current_term: 1 leader_uuid: "75525021a76043b8a1227ca817b38ec8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7232cd23c889418689f2c886ba008313" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 35961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "75525021a76043b8a1227ca817b38ec8" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "519801f0627c402c9a65ccab106caf51" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 33791 } health_report { overall_health: UNKNOWN } } }
I20260620 18:36:26.277491 30159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:26.272271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55912 (local address 127.22.37.3:35311)
0620 18:36:26.272494 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:26.272501 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:26.272520 (+ 19us) server_negotiation.cc:407] Connection header received
0620 18:36:26.272746 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:26.272750 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:26.272835 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:26.272990 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:36:26.273394 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:26.274100 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:26.275367 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:26.275784 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:26.275913 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:26.276332 (+ 419us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:36:26.276874 (+ 542us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:36:26.277032 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:26.277088 (+ 56us) server_negotiation.cc:299] Negotiation successful
0620 18:36:26.277163 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 18:36:26.281281 29732 raft_consensus.cc:1261] T 199986f855a849ebb076e6dde8df6f83 P 7232cd23c889418689f2c886ba008313 [term 1 FOLLOWER]: Refusing update from remote peer 75525021a76043b8a1227ca817b38ec8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:36:26.281625 29877 raft_consensus.cc:1261] T 199986f855a849ebb076e6dde8df6f83 P 519801f0627c402c9a65ccab106caf51 [term 1 FOLLOWER]: Refusing update from remote peer 75525021a76043b8a1227ca817b38ec8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:36:26.282059 30167 consensus_queue.cc:1035] T 199986f855a849ebb076e6dde8df6f83 P 75525021a76043b8a1227ca817b38ec8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7232cd23c889418689f2c886ba008313" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 35961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:36:26.282382 30172 consensus_queue.cc:1035] T 199986f855a849ebb076e6dde8df6f83 P 75525021a76043b8a1227ca817b38ec8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "519801f0627c402c9a65ccab106caf51" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 33791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:36:26.285166 30177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:26.284579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:46099 (local address 127.22.37.1:60649)
0620 18:36:26.284818 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:26.284927 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:46099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
I20260620 18:36:26.285543 30173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:26.284915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:46099 (local address 127.22.37.2:42909)
0620 18:36:26.285123 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:26.285305 (+ 182us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:46099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260620 18:36:26.291695 30173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:26.290540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:46099 (local address 127.22.37.2:40709)
0620 18:36:26.291439 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:26.291526 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:46099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":772}
I20260620 18:36:26.295584 30177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:26.295025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:46099 (local address 127.22.37.1:43133)
0620 18:36:26.295282 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:26.295389 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:46099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
I20260620 18:36:26.299587 30170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:26.299019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:46099 (local address 127.22.37.3:39915)
0620 18:36:26.299316 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:26.299410 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:46099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190}
I20260620 18:36:26.309023 29416 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41210:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:36:26.309325 29416 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41210:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:36:26.314953 29416 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b1f845ed42fb4dcd90063c45dedac043: Sending DeleteTablet for 3 replicas of tablet 199986f855a849ebb076e6dde8df6f83
I20260620 18:36:26.316082 29857 tablet_service.cc:1514] Processing DeleteTablet for tablet 199986f855a849ebb076e6dde8df6f83 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:36:26 UTC) from {username='oryx', principal='oryx/127.22.37.61@KRBTEST.COM'} at 127.0.0.1:56094
I20260620 18:36:26.316123 29712 tablet_service.cc:1514] Processing DeleteTablet for tablet 199986f855a849ebb076e6dde8df6f83 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:36:26 UTC) from {username='oryx', principal='oryx/127.22.37.61@KRBTEST.COM'} at 127.0.0.1:36634
I20260620 18:36:26.316816 29998 tablet_service.cc:1514] Processing DeleteTablet for tablet 199986f855a849ebb076e6dde8df6f83 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:36:26 UTC) from {username='oryx', principal='oryx/127.22.37.61@KRBTEST.COM'} at 127.0.0.1:55904
I20260620 18:36:26.317243 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 29624
I20260620 18:36:26.322582 30192 tablet_replica.cc:330] T 199986f855a849ebb076e6dde8df6f83 P 75525021a76043b8a1227ca817b38ec8: stopping tablet replica
I20260620 18:36:26.322774 30192 raft_consensus.cc:2229] T 199986f855a849ebb076e6dde8df6f83 P 75525021a76043b8a1227ca817b38ec8 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:36:26.323339 30192 raft_consensus.cc:2258] T 199986f855a849ebb076e6dde8df6f83 P 75525021a76043b8a1227ca817b38ec8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:36:26.325865 30191 tablet_replica.cc:330] T 199986f855a849ebb076e6dde8df6f83 P 519801f0627c402c9a65ccab106caf51: stopping tablet replica
I20260620 18:36:26.326063 30191 raft_consensus.cc:2229] T 199986f855a849ebb076e6dde8df6f83 P 519801f0627c402c9a65ccab106caf51 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:36:26.326371 30191 raft_consensus.cc:2258] T 199986f855a849ebb076e6dde8df6f83 P 519801f0627c402c9a65ccab106caf51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:36:26.327716 30190 tablet_replica.cc:330] T 199986f855a849ebb076e6dde8df6f83 P 7232cd23c889418689f2c886ba008313: stopping tablet replica
I20260620 18:36:26.328254 30192 ts_tablet_manager.cc:1905] T 199986f855a849ebb076e6dde8df6f83 P 75525021a76043b8a1227ca817b38ec8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:36:26.327910 30190 raft_consensus.cc:2229] T 199986f855a849ebb076e6dde8df6f83 P 7232cd23c889418689f2c886ba008313 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:36:26.330997 30190 raft_consensus.cc:2258] T 199986f855a849ebb076e6dde8df6f83 P 7232cd23c889418689f2c886ba008313 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:36:26.332197 30190 ts_tablet_manager.cc:1905] T 199986f855a849ebb076e6dde8df6f83 P 7232cd23c889418689f2c886ba008313: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:36:26.332389 30191 ts_tablet_manager.cc:1905] T 199986f855a849ebb076e6dde8df6f83 P 519801f0627c402c9a65ccab106caf51: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:36:26.340052 30190 ts_tablet_manager.cc:1918] T 199986f855a849ebb076e6dde8df6f83 P 7232cd23c889418689f2c886ba008313: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:36:26.340298 30190 log.cc:1196] T 199986f855a849ebb076e6dde8df6f83 P 7232cd23c889418689f2c886ba008313: Deleting WAL directory at /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/ts-0/wal/wals/199986f855a849ebb076e6dde8df6f83
I20260620 18:36:26.340862 30190 ts_tablet_manager.cc:1939] T 199986f855a849ebb076e6dde8df6f83 P 7232cd23c889418689f2c886ba008313: Deleting consensus metadata
I20260620 18:36:26.341670 30192 ts_tablet_manager.cc:1918] T 199986f855a849ebb076e6dde8df6f83 P 75525021a76043b8a1227ca817b38ec8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:36:26.341823 30192 log.cc:1196] T 199986f855a849ebb076e6dde8df6f83 P 75525021a76043b8a1227ca817b38ec8: Deleting WAL directory at /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/ts-2/wal/wals/199986f855a849ebb076e6dde8df6f83
I20260620 18:36:26.342360 30192 ts_tablet_manager.cc:1939] T 199986f855a849ebb076e6dde8df6f83 P 75525021a76043b8a1227ca817b38ec8: Deleting consensus metadata
I20260620 18:36:26.342587 29402 catalog_manager.cc:4841] TS 7232cd23c889418689f2c886ba008313 (127.22.37.1:35961): tablet 199986f855a849ebb076e6dde8df6f83 (table test-table [id=6e36f8d4312d471dada837d9ba8b47bc]) successfully deleted
I20260620 18:36:26.343187 30191 ts_tablet_manager.cc:1918] T 199986f855a849ebb076e6dde8df6f83 P 519801f0627c402c9a65ccab106caf51: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:36:26.343329 30191 log.cc:1196] T 199986f855a849ebb076e6dde8df6f83 P 519801f0627c402c9a65ccab106caf51: Deleting WAL directory at /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980527431848-22676-0/minicluster-data/ts-1/wal/wals/199986f855a849ebb076e6dde8df6f83
I20260620 18:36:26.343538 29401 catalog_manager.cc:4841] TS 75525021a76043b8a1227ca817b38ec8 (127.22.37.3:35311): tablet 199986f855a849ebb076e6dde8df6f83 (table test-table [id=6e36f8d4312d471dada837d9ba8b47bc]) successfully deleted
I20260620 18:36:26.343788 30191 ts_tablet_manager.cc:1939] T 199986f855a849ebb076e6dde8df6f83 P 519801f0627c402c9a65ccab106caf51: Deleting consensus metadata
I20260620 18:36:26.344759 29403 catalog_manager.cc:4841] TS 519801f0627c402c9a65ccab106caf51 (127.22.37.2:33791): tablet 199986f855a849ebb076e6dde8df6f83 (table test-table [id=6e36f8d4312d471dada837d9ba8b47bc]) successfully deleted
I20260620 18:36:26.347424 30158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:26.318213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36650 (local address 127.22.37.1:35961)
0620 18:36:26.318422 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:26.318430 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:26.322923 (+ 4493us) server_negotiation.cc:407] Connection header received
0620 18:36:26.323044 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:26.323053 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:26.323157 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:26.323280 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:26.325074 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:26.327329 (+ 2255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:26.331852 (+ 4523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:26.332457 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:26.332580 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:26.336762 (+ 4182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:26.336781 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:26.336802 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:26.336849 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:26.341040 (+ 4191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:26.345220 (+ 4180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:26.345231 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:26.345234 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:26.345423 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:26.345815 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:26.345820 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:26.345824 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:26.346197 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:26.346485 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:26.346845 (+ 360us) server_negotiation.cc:299] Negotiation successful
0620 18:36:26.347111 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 18:36:26.348212 29772 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:36:26.442140 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 29624
I20260620 18:36:26.460392 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 29794
I20260620 18:36:26.475570 30161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:26.461279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56108 (local address 127.22.37.2:33791)
0620 18:36:26.461513 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:26.461520 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:26.461562 (+ 42us) server_negotiation.cc:407] Connection header received
0620 18:36:26.461752 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:26.461758 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:26.461848 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:26.461968 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:26.463278 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:26.464406 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:26.465704 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:26.466327 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:26.466453 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:26.468907 (+ 2454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:26.468926 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:26.468947 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:26.468996 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:26.472739 (+ 3743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:26.473348 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:26.473357 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:26.473361 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:26.473530 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:26.473972 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:26.473976 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:26.473980 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:26.474429 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:26.474732 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:26.474990 (+ 258us) server_negotiation.cc:299] Negotiation successful
0620 18:36:26.475203 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260620 18:36:26.476317 29917 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:36:26.595790 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 29794
I20260620 18:36:26.625203 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 29935
I20260620 18:36:26.641515 30159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:26.626097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55920 (local address 127.22.37.3:35311)
0620 18:36:26.626271 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:26.626278 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:26.626333 (+ 55us) server_negotiation.cc:407] Connection header received
0620 18:36:26.626552 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:26.626566 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:26.626629 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:26.626762 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:26.628237 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:26.629676 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:26.630945 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:26.631639 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:26.631774 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:26.634230 (+ 2456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:26.634249 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:26.634276 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:26.634326 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:26.638617 (+ 4291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:26.639251 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:26.639261 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:26.639264 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:26.639451 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:26.639834 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:26.639839 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:26.639843 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:26.640300 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:26.640632 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:26.640828 (+ 196us) server_negotiation.cc:299] Negotiation successful
0620 18:36:26.641058 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 18:36:26.642328 30058 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:36:26.735431 30197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:26.734769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:46099 (local address 127.0.0.1:44478)
0620 18:36:26.735175 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:26.735274 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:46099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":218,"threads_started":1}
I20260620 18:36:26.736733 30197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:26.736393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:46099 (local address 127.0.0.1:44486)
0620 18:36:26.736544 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:26.736631 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:46099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260620 18:36:26.770466 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 29935
I20260620 18:36:26.791714 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 29272
I20260620 18:36:26.792718 29446 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:36:26.915594 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 29272
I20260620 18:36:26.936914 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 29375
I20260620 18:36:26.937956 29629 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:36:27.051497 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 29375
2026-06-20T18:36:27Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9454 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:36:27 dist-test-slave-5lgp krb5kdc[30203](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:36:27 dist-test-slave-5lgp krb5kdc[30203](info): set up 2 sockets
Jun 20 18:36:27 dist-test-slave-5lgp krb5kdc[30203](info): commencing operation
krb5kdc: starting...
W20260620 18:36:28.162357 22676 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.002s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/krb5kdc/test-user.keytab.
Jun 20 18:36:28 dist-test-slave-5lgp krb5kdc[30203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980588, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:36:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:36:28Z Disabled control of system clock
WARNING: no policy specified for oryx/127.22.37.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.37.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.37.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/krb5kdc/oryx_127.22.37.62.keytab.
Entry for principal oryx/127.22.37.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/krb5kdc/oryx_127.22.37.62.keytab.
I20260620 18:36:28.368997 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.62:42963
--webserver_interface=127.22.37.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/krb5kdc/oryx_127.22.37.62.keytab
--principal=oryx/127.22.37.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:33605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.37.62:42963
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:36:28.681761 30216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:36:28.682104 30216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:36:28.682283 30216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:36:28.689318 30216 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:36:28.689472 30216 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 18:36:28.689566 30216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:36:28.689684 30216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:36:28.689769 30216 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:36:28.689857 30216 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:36:28.699502 30216 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:33605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.37.62:42963
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/krb5kdc/oryx_127.22.37.62.keytab
--principal=oryx/127.22.37.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.62:42963
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.37.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:36:28.700815 30216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:36:28.702167 30216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:36:28.713866 30224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:28.713936 30221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:28.713948 30222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:28.714612 30216 server_base.cc:1034] running on GCE node
I20260620 18:36:28.715386 30216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:36:28.716715 30216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:36:28.717954 30216 hybrid_clock.cc:648] HybridClock initialized: now 1781980588717895 us; error 39 us; skew 500 ppm
I20260620 18:36:28.718277 30216 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:36:28 dist-test-slave-5lgp krb5kdc[30203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980588, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.37.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:36:28.721323 30216 init.cc:377] Logged in from keytab as oryx/127.22.37.62@KRBTEST.COM (short username oryx)
I20260620 18:36:28.722811 30216 webserver.cc:460] Webserver started at http://127.22.37.62:42323/ using document root <none> and password file <none>
I20260620 18:36:28.723506 30216 fs_manager.cc:359] Metadata directory not provided
I20260620 18:36:28.723630 30216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:36:28.723963 30216 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:36:28.726879 30216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/minicluster-data/master-0/data/instance:
uuid: "4d9466ac6e3e4d1c95493d9373c03f45"
format_stamp: "Formatted at 2026-06-20 18:36:28 on dist-test-slave-5lgp"
server_key: "8390940272300e6550b8b1a68d86d4aa"
server_key_iv: "3a1171a75fb3105d32b6ddc1e3dabefd"
server_key_version: "encryptionkey@0"
I20260620 18:36:28.727632 30216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/minicluster-data/master-0/wal/instance:
uuid: "4d9466ac6e3e4d1c95493d9373c03f45"
format_stamp: "Formatted at 2026-06-20 18:36:28 on dist-test-slave-5lgp"
server_key: "8390940272300e6550b8b1a68d86d4aa"
server_key_iv: "3a1171a75fb3105d32b6ddc1e3dabefd"
server_key_version: "encryptionkey@0"
I20260620 18:36:28.732450 30216 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 18:36:28.735772 30231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:28.737053 30216 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:36:28.737224 30216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/minicluster-data/master-0/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/minicluster-data/master-0/wal
uuid: "4d9466ac6e3e4d1c95493d9373c03f45"
format_stamp: "Formatted at 2026-06-20 18:36:28 on dist-test-slave-5lgp"
server_key: "8390940272300e6550b8b1a68d86d4aa"
server_key_iv: "3a1171a75fb3105d32b6ddc1e3dabefd"
server_key_version: "encryptionkey@0"
I20260620 18:36:28.737450 30216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:36:28.750133 30216 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:36:28.754047 30216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:36:28.754321 30216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:36:28.775702 30216 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.62:42963
I20260620 18:36:28.775702 30292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.62:42963 every 8 connection(s)
I20260620 18:36:28.777077 30216 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
I20260620 18:36:28.778841 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 30216
I20260620 18:36:28.779047 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/minicluster-data/master-0/wal/instance
I20260620 18:36:28.779881 22676 external_mini_cluster.cc:1428] Setting key a9babe28581a244f7a929b8ca7acfe80
I20260620 18:36:28.781810 30293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 18:36:28 dist-test-slave-5lgp krb5kdc[30203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980588, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM
I20260620 18:36:28.795310 30293 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d9466ac6e3e4d1c95493d9373c03f45: Bootstrap starting.
I20260620 18:36:28.798770 30293 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d9466ac6e3e4d1c95493d9373c03f45: Neither blocks nor log segments found. Creating new log.
I20260620 18:36:28.800005 30293 log.cc:824] T 00000000000000000000000000000000 P 4d9466ac6e3e4d1c95493d9373c03f45: Log is configured to *not* fsync() on all Append() calls
I20260620 18:36:28.802939 30296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:28.781943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43568 (local address 127.22.37.62:42963)
0620 18:36:28.782686 (+ 743us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:28.782704 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:28.782742 (+ 38us) server_negotiation.cc:407] Connection header received
0620 18:36:28.784241 (+ 1499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:28.784261 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:28.784635 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:28.784960 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:28.786688 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:28.787991 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:28.789746 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:28.790382 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:28.790509 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:28.795355 (+ 4846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:28.795363 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:28.795402 (+ 39us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:28.795453 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:28.798476 (+ 3023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:28.799305 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:28.799311 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:28.799317 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:28.799473 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:28.799875 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:28.799878 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:28.799880 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:28.800410 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:28.800798 (+ 388us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:28.801197 (+ 399us) server_negotiation.cc:299] Negotiation successful
0620 18:36:28.801650 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":349,"threads_started":1}
I20260620 18:36:28.803396 30293 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d9466ac6e3e4d1c95493d9373c03f45: No bootstrap required, opened a new log
I20260620 18:36:28.811137 30293 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d9466ac6e3e4d1c95493d9373c03f45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d9466ac6e3e4d1c95493d9373c03f45" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 42963 } }
I20260620 18:36:28.811480 30293 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4d9466ac6e3e4d1c95493d9373c03f45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:36:28.811550 30293 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d9466ac6e3e4d1c95493d9373c03f45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d9466ac6e3e4d1c95493d9373c03f45, State: Initialized, Role: FOLLOWER
I20260620 18:36:28.812018 30293 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d9466ac6e3e4d1c95493d9373c03f45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d9466ac6e3e4d1c95493d9373c03f45" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 42963 } }
I20260620 18:36:28.812214 30293 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4d9466ac6e3e4d1c95493d9373c03f45 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:36:28.812309 30293 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d9466ac6e3e4d1c95493d9373c03f45 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:36:28.812426 30293 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d9466ac6e3e4d1c95493d9373c03f45 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:36:28.814481 30293 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d9466ac6e3e4d1c95493d9373c03f45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d9466ac6e3e4d1c95493d9373c03f45" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 42963 } }
I20260620 18:36:28.815068 30293 leader_election.cc:302] T 00000000000000000000000000000000 P 4d9466ac6e3e4d1c95493d9373c03f45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d9466ac6e3e4d1c95493d9373c03f45; no voters:
I20260620 18:36:28.815726 30293 leader_election.cc:288] T 00000000000000000000000000000000 P 4d9466ac6e3e4d1c95493d9373c03f45 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:36:28.815864 30298 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d9466ac6e3e4d1c95493d9373c03f45 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:36:28.816583 30298 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d9466ac6e3e4d1c95493d9373c03f45 [term 1 LEADER]: Becoming Leader. State: Replica: 4d9466ac6e3e4d1c95493d9373c03f45, State: Running, Role: LEADER
I20260620 18:36:28.817129 30298 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d9466ac6e3e4d1c95493d9373c03f45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d9466ac6e3e4d1c95493d9373c03f45" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 42963 } }
I20260620 18:36:28.817833 30293 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d9466ac6e3e4d1c95493d9373c03f45 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:36:28.820924 30299 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d9466ac6e3e4d1c95493d9373c03f45 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4d9466ac6e3e4d1c95493d9373c03f45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d9466ac6e3e4d1c95493d9373c03f45" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 42963 } } }
I20260620 18:36:28.821250 30299 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d9466ac6e3e4d1c95493d9373c03f45 [sys.catalog]: This master's current role is: LEADER
I20260620 18:36:28.822049 30300 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d9466ac6e3e4d1c95493d9373c03f45 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d9466ac6e3e4d1c95493d9373c03f45. Latest consensus state: current_term: 1 leader_uuid: "4d9466ac6e3e4d1c95493d9373c03f45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d9466ac6e3e4d1c95493d9373c03f45" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 42963 } } }
I20260620 18:36:28.822453 30300 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d9466ac6e3e4d1c95493d9373c03f45 [sys.catalog]: This master's current role is: LEADER
I20260620 18:36:28.822714 30307 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:36:28.828033 30307 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:36:28.837656 30307 catalog_manager.cc:1347] Generated new cluster ID: be7ecfd3512e417897f4def456e19ff0
I20260620 18:36:28.837787 30307 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:36:28.852484 30307 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:36:28.853821 30307 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:36:28.862025 30307 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4d9466ac6e3e4d1c95493d9373c03f45: Generated new TSK 0
I20260620 18:36:28.862860 30307 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.22.37.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.37.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.37.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/krb5kdc/oryx_127.22.37.1.keytab.
Entry for principal oryx/127.22.37.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/krb5kdc/oryx_127.22.37.1.keytab.
I20260620 18:36:28.916934 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.37.1:0
--local_ip_for_outbound_sockets=127.22.37.1
--webserver_interface=127.22.37.1
--webserver_port=0
--tserver_master_addrs=127.22.37.62:42963
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/krb5kdc/oryx_127.22.37.1.keytab
--principal=oryx/127.22.37.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:33605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:36:29.243743 30319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:36:29.244117 30319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:36:29.244298 30319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:36:29.251573 30319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:36:29.251679 30319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:36:29.251848 30319 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 18:36:29.251979 30319 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.37.1
I20260620 18:36:29.261579 30319 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:33605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/krb5kdc/oryx_127.22.37.1.keytab
--principal=oryx/127.22.37.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.37.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.37.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.37.62:42963
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.37.1
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:36:29.262904 30319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:36:29.264261 30319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:36:29.277563 30324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:29.277599 30327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:29.277623 30325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:29.278366 30319 server_base.cc:1034] running on GCE node
I20260620 18:36:29.278986 30319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:36:29.279843 30319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:36:29.281070 30319 hybrid_clock.cc:648] HybridClock initialized: now 1781980589281010 us; error 34 us; skew 500 ppm
I20260620 18:36:29.281462 30319 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:36:29 dist-test-slave-5lgp krb5kdc[30203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980589, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.37.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:36:29.284569 30319 init.cc:377] Logged in from keytab as oryx/127.22.37.1@KRBTEST.COM (short username oryx)
I20260620 18:36:29.286123 30319 webserver.cc:460] Webserver started at http://127.22.37.1:36431/ using document root <none> and password file <none>
I20260620 18:36:29.286765 30319 fs_manager.cc:359] Metadata directory not provided
I20260620 18:36:29.286849 30319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:36:29.287127 30319 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:36:29.290017 30319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/minicluster-data/ts-0/data/instance:
uuid: "b94701dfaaea4c6c95f229005e7b613e"
format_stamp: "Formatted at 2026-06-20 18:36:29 on dist-test-slave-5lgp"
server_key: "ac4763ce403818ececb1b0c28b6cc533"
server_key_iv: "ee39cd4a552bd82d4430f65f6a9f41c4"
server_key_version: "encryptionkey@0"
I20260620 18:36:29.290759 30319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/minicluster-data/ts-0/wal/instance:
uuid: "b94701dfaaea4c6c95f229005e7b613e"
format_stamp: "Formatted at 2026-06-20 18:36:29 on dist-test-slave-5lgp"
server_key: "ac4763ce403818ececb1b0c28b6cc533"
server_key_iv: "ee39cd4a552bd82d4430f65f6a9f41c4"
server_key_version: "encryptionkey@0"
I20260620 18:36:29.296110 30319 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260620 18:36:29.299808 30334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:29.301167 30319 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:36:29.301355 30319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/minicluster-data/ts-0/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/minicluster-data/ts-0/wal
uuid: "b94701dfaaea4c6c95f229005e7b613e"
format_stamp: "Formatted at 2026-06-20 18:36:29 on dist-test-slave-5lgp"
server_key: "ac4763ce403818ececb1b0c28b6cc533"
server_key_iv: "ee39cd4a552bd82d4430f65f6a9f41c4"
server_key_version: "encryptionkey@0"
I20260620 18:36:29.301674 30319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:36:29.326886 30319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:36:29.331439 30319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:36:29.331717 30319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:36:29.334431 30319 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:36:29.334542 30319 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:29.334733 30319 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:36:29.334816 30319 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 18:36:29 dist-test-slave-5lgp krb5kdc[30203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980589, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.37.1@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM
I20260620 18:36:29.357972 30343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:29.334930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.1:59157 (local address 127.22.37.62:42963)
0620 18:36:29.335717 (+ 787us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:29.335724 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:29.336758 (+ 1034us) server_negotiation.cc:407] Connection header received
0620 18:36:29.338877 (+ 2119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:29.338882 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:29.338977 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:29.339098 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:29.341280 (+ 2182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:29.342262 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:29.344304 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:29.344885 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:29.345033 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:29.351308 (+ 6275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:29.351319 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:29.351326 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:29.351371 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:29.354825 (+ 3454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:29.355586 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:29.355593 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:29.355597 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:29.355738 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:29.356170 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:29.356176 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:29.356180 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:29.356460 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:29.356610 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:29.357321 (+ 711us) server_negotiation.cc:299] Negotiation successful
0620 18:36:29.357591 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":646,"thread_start_us":539,"threads_started":1}
I20260620 18:36:29.359107 30344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:29.335689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:42963 (local address 127.22.37.1:59157)
0620 18:36:29.336566 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:29.336622 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:29.338602 (+ 1980us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:29.339379 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:29.339393 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:29.340235 (+ 842us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:29.341105 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:29.341124 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:29.342432 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:29.342437 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:29.344026 (+ 1589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:29.344038 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:29.345171 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:29.345187 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:29.345692 (+ 505us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:29.346813 (+ 1121us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:29.346858 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:29.350920 (+ 4062us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:36:29.354972 (+ 4052us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:29.354981 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:29.355007 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:29.355458 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:29.355848 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:29.355854 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:29.355859 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:29.356056 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:29.356585 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:36:29.356593 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:36:29.356952 (+ 359us) client_negotiation.cc:769] Sending connection context
0620 18:36:29.357247 (+ 295us) client_negotiation.cc:240] Negotiation successful
0620 18:36:29.357871 (+ 624us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":386,"threads_started":1}
I20260620 18:36:29.383987 30343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:29.367752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.1:33455 (local address 127.22.37.62:42963)
0620 18:36:29.367964 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:29.367974 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:29.368839 (+ 865us) server_negotiation.cc:407] Connection header received
0620 18:36:29.369052 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:29.369061 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:29.369157 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:29.369291 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:29.370444 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:29.371684 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:29.372984 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:29.373796 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:29.373943 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:29.376815 (+ 2872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:29.376822 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:29.376830 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:29.376881 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:29.380568 (+ 3687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:29.381592 (+ 1024us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:29.381598 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:29.381603 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:29.381778 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:29.382289 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:29.382296 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:29.382301 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:29.382581 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:29.382691 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:29.383346 (+ 655us) server_negotiation.cc:299] Negotiation successful
0620 18:36:29.383613 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 18:36:29.384402 30405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:29.367881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:42963 (local address 127.22.37.1:33455)
0620 18:36:29.368710 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:29.368730 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:29.368905 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:29.369331 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:29.369338 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:29.369997 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:29.370282 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:29.370291 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:29.371818 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:29.371822 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:29.372843 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:29.372854 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:29.374038 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:29.374050 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:29.374309 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:29.375170 (+ 861us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:29.375212 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:29.376596 (+ 1384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:36:29.380730 (+ 4134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:29.380738 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:29.380748 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:29.381465 (+ 717us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:29.381917 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:29.381925 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:29.381930 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:29.382136 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:29.382705 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:36:29.382711 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:36:29.382989 (+ 278us) client_negotiation.cc:769] Sending connection context
0620 18:36:29.383181 (+ 192us) client_negotiation.cc:240] Negotiation successful
0620 18:36:29.383665 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":673,"thread_start_us":544,"threads_started":1}
I20260620 18:36:29.389235 30319 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.1:44231
I20260620 18:36:29.389324 30454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.1:44231 every 8 connection(s)
I20260620 18:36:29.390473 30319 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/minicluster-data/ts-0/data/info.pb
I20260620 18:36:29.392393 30455 heartbeater.cc:347] Connected to a master server at 127.22.37.62:42963
I20260620 18:36:29.392625 30455 heartbeater.cc:464] Registering TS with master...
I20260620 18:36:29.393236 30455 heartbeater.cc:511] Master 127.22.37.62:42963 requested a full tablet report, sending...
I20260620 18:36:29.395146 30248 ts_manager.cc:194] Registered new tserver with Master: b94701dfaaea4c6c95f229005e7b613e (127.22.37.1:44231)
I20260620 18:36:29.396613 30248 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.37.1@KRBTEST.COM'} at 127.22.37.1:59157
I20260620 18:36:29.399763 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 30319
I20260620 18:36:29.399956 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/minicluster-data/ts-0/wal/instance
I20260620 18:36:29.400609 22676 external_mini_cluster.cc:1428] Setting key 866d49e46a1232c6c69b9ae8a146ef19
WARNING: no policy specified for oryx/127.22.37.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.37.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.37.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/krb5kdc/oryx_127.22.37.2.keytab.
Entry for principal oryx/127.22.37.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/krb5kdc/oryx_127.22.37.2.keytab.
I20260620 18:36:29.445195 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.37.2:0
--local_ip_for_outbound_sockets=127.22.37.2
--webserver_interface=127.22.37.2
--webserver_port=0
--tserver_master_addrs=127.22.37.62:42963
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/krb5kdc/oryx_127.22.37.2.keytab
--principal=oryx/127.22.37.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:33605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:36:29.761504 30459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:36:29.761888 30459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:36:29.762053 30459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:36:29.769129 30459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:36:29.769232 30459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:36:29.769445 30459 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 18:36:29.769570 30459 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.37.2
I20260620 18:36:29.779086 30459 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:33605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/krb5kdc/oryx_127.22.37.2.keytab
--principal=oryx/127.22.37.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.37.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.37.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.37.62:42963
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.37.2
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:36:29.780287 30459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:36:29.781625 30459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:36:29.794590 30464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:29.794520 30465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:29.794538 30467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:29.795796 30459 server_base.cc:1034] running on GCE node
I20260620 18:36:29.796445 30459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:36:29.797358 30459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:36:29.798645 30459 hybrid_clock.cc:648] HybridClock initialized: now 1781980589798579 us; error 39 us; skew 500 ppm
I20260620 18:36:29.798945 30459 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:36:29 dist-test-slave-5lgp krb5kdc[30203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980589, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.37.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:36:29.805962 30459 init.cc:377] Logged in from keytab as oryx/127.22.37.2@KRBTEST.COM (short username oryx)
I20260620 18:36:29.807410 30459 webserver.cc:460] Webserver started at http://127.22.37.2:35137/ using document root <none> and password file <none>
I20260620 18:36:29.808080 30459 fs_manager.cc:359] Metadata directory not provided
I20260620 18:36:29.808208 30459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:36:29.808533 30459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:36:29.811407 30459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/minicluster-data/ts-1/data/instance:
uuid: "856016baca2c4cf3b35c8b1c3d8ac989"
format_stamp: "Formatted at 2026-06-20 18:36:29 on dist-test-slave-5lgp"
server_key: "d04a2aa10e75d6a322b638967994ab5d"
server_key_iv: "d9fcc5a9aa63d45c4698a7310246c449"
server_key_version: "encryptionkey@0"
I20260620 18:36:29.812177 30459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/minicluster-data/ts-1/wal/instance:
uuid: "856016baca2c4cf3b35c8b1c3d8ac989"
format_stamp: "Formatted at 2026-06-20 18:36:29 on dist-test-slave-5lgp"
server_key: "d04a2aa10e75d6a322b638967994ab5d"
server_key_iv: "d9fcc5a9aa63d45c4698a7310246c449"
server_key_version: "encryptionkey@0"
I20260620 18:36:29.817188 30459 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260620 18:36:29.821000 30474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:29.822443 30459 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:36:29.822649 30459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/minicluster-data/ts-1/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/minicluster-data/ts-1/wal
uuid: "856016baca2c4cf3b35c8b1c3d8ac989"
format_stamp: "Formatted at 2026-06-20 18:36:29 on dist-test-slave-5lgp"
server_key: "d04a2aa10e75d6a322b638967994ab5d"
server_key_iv: "d9fcc5a9aa63d45c4698a7310246c449"
server_key_version: "encryptionkey@0"
I20260620 18:36:29.822877 30459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:36:29.842666 30459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:36:29.847227 30459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:36:29.847513 30459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:36:29.850180 30459 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:36:29.850291 30459 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:29.850503 30459 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:36:29.850634 30459 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Jun 20 18:36:29 dist-test-slave-5lgp krb5kdc[30203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980589, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.37.2@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM
I20260620 18:36:29.873958 30343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:29.850708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.2:52567 (local address 127.22.37.62:42963)
0620 18:36:29.850904 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:29.850910 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:29.852272 (+ 1362us) server_negotiation.cc:407] Connection header received
0620 18:36:29.854487 (+ 2215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:29.854496 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:29.854644 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:29.854764 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:29.856993 (+ 2229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:29.857976 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:29.861300 (+ 3324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:29.861842 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:29.861998 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:29.866228 (+ 4230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:29.866233 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:29.866236 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:29.866267 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:29.870285 (+ 4018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:29.871310 (+ 1025us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:29.871318 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:29.871324 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:29.871487 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:29.871963 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:29.871975 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:29.871978 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:29.872379 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:29.872550 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:29.873387 (+ 837us) server_negotiation.cc:299] Negotiation successful
0620 18:36:29.873602 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 18:36:29.874950 30483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:29.851205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:42963 (local address 127.22.37.2:52567)
0620 18:36:29.852079 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:29.852128 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:29.854159 (+ 2031us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:29.855048 (+ 889us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:29.855059 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:29.855850 (+ 791us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:29.856799 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:29.856825 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:29.858189 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:29.858197 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:29.861075 (+ 2878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:29.861120 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:29.862117 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:29.862127 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:29.862413 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:29.863140 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:29.863164 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:29.865902 (+ 2738us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:36:29.870450 (+ 4548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:29.870462 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:29.870484 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:29.871172 (+ 688us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:29.871622 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:29.871625 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:29.871628 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:29.871813 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:29.872641 (+ 828us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:36:29.872649 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:36:29.872995 (+ 346us) client_negotiation.cc:769] Sending connection context
0620 18:36:29.873269 (+ 274us) client_negotiation.cc:240] Negotiation successful
0620 18:36:29.873828 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":368,"threads_started":1}
I20260620 18:36:29.894055 30343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:29.880499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.2:59591 (local address 127.22.37.62:42963)
0620 18:36:29.880713 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:29.880719 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:29.881364 (+ 645us) server_negotiation.cc:407] Connection header received
0620 18:36:29.881587 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:29.881591 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:29.881649 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:29.881765 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:29.883082 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:29.884128 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:29.885630 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:29.886033 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:29.886171 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:29.887899 (+ 1728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:29.887910 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:29.887913 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:29.887942 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:29.890893 (+ 2951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:29.891702 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:29.891710 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:29.891716 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:29.891893 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:29.892335 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:29.892343 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:29.892347 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:29.892856 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:29.893008 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:29.893487 (+ 479us) server_negotiation.cc:299] Negotiation successful
0620 18:36:29.893702 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 18:36:29.894311 30537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:29.880476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:42963 (local address 127.22.37.2:59591)
0620 18:36:29.881244 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:29.881266 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:29.881458 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:29.881801 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:29.881809 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:29.882305 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:29.882944 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:29.882958 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:29.884262 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:29.884268 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:29.885477 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:29.885491 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:29.886212 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:29.886218 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:29.886400 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:29.887009 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:29.887025 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:29.887695 (+ 670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:36:29.891023 (+ 3328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:29.891030 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:29.891035 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:29.891565 (+ 530us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:29.892020 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:29.892023 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:29.892026 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:29.892206 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:29.892973 (+ 767us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:36:29.892978 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:36:29.893170 (+ 192us) client_negotiation.cc:769] Sending connection context
0620 18:36:29.893387 (+ 217us) client_negotiation.cc:240] Negotiation successful
0620 18:36:29.893743 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":382,"threads_started":1}
I20260620 18:36:29.908157 30459 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.2:46153
I20260620 18:36:29.908197 30593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.2:46153 every 8 connection(s)
I20260620 18:36:29.909483 30459 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/minicluster-data/ts-1/data/info.pb
I20260620 18:36:29.911871 30594 heartbeater.cc:347] Connected to a master server at 127.22.37.62:42963
I20260620 18:36:29.912093 30594 heartbeater.cc:464] Registering TS with master...
I20260620 18:36:29.912806 30594 heartbeater.cc:511] Master 127.22.37.62:42963 requested a full tablet report, sending...
I20260620 18:36:29.914382 30248 ts_manager.cc:194] Registered new tserver with Master: 856016baca2c4cf3b35c8b1c3d8ac989 (127.22.37.2:46153)
I20260620 18:36:29.915243 30248 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.37.2@KRBTEST.COM'} at 127.22.37.2:52567
I20260620 18:36:29.915253 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 30459
I20260620 18:36:29.915484 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/minicluster-data/ts-1/wal/instance
I20260620 18:36:29.916060 22676 external_mini_cluster.cc:1428] Setting key fa60008b245ffc89089c12bc53be8177
WARNING: no policy specified for oryx/127.22.37.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.37.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.37.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/krb5kdc/oryx_127.22.37.3.keytab.
Entry for principal oryx/127.22.37.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/krb5kdc/oryx_127.22.37.3.keytab.
I20260620 18:36:29.960896 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.37.3:0
--local_ip_for_outbound_sockets=127.22.37.3
--webserver_interface=127.22.37.3
--webserver_port=0
--tserver_master_addrs=127.22.37.62:42963
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/krb5kdc/oryx_127.22.37.3.keytab
--principal=oryx/127.22.37.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:33605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:36:30.283227 30598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:36:30.283531 30598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:36:30.283696 30598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:36:30.291167 30598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:36:30.291283 30598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:36:30.291457 30598 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 18:36:30.291548 30598 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.37.3
I20260620 18:36:30.301553 30598 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:33605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/krb5kdc/oryx_127.22.37.3.keytab
--principal=oryx/127.22.37.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.37.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.37.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.37.62:42963
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.37.3
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:36:30.302906 30598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:36:30.304235 30598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:36:30.317677 30606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:30.317663 30604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:30.318270 30603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:30.318639 30598 server_base.cc:1034] running on GCE node
I20260620 18:36:30.319373 30598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:36:30.320468 30598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:36:30.321700 30598 hybrid_clock.cc:648] HybridClock initialized: now 1781980590321637 us; error 40 us; skew 500 ppm
I20260620 18:36:30.322016 30598 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:36:30 dist-test-slave-5lgp krb5kdc[30203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980590, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.37.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:36:30.325249 30598 init.cc:377] Logged in from keytab as oryx/127.22.37.3@KRBTEST.COM (short username oryx)
I20260620 18:36:30.326845 30598 webserver.cc:460] Webserver started at http://127.22.37.3:33967/ using document root <none> and password file <none>
I20260620 18:36:30.327639 30598 fs_manager.cc:359] Metadata directory not provided
I20260620 18:36:30.327734 30598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:36:30.327996 30598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:36:30.330953 30598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/minicluster-data/ts-2/data/instance:
uuid: "622c643eaee54366b09419fb11891fe9"
format_stamp: "Formatted at 2026-06-20 18:36:30 on dist-test-slave-5lgp"
server_key: "3399f6cd32653191f7bcee6011606f87"
server_key_iv: "0a9a8d09fb3770c35b0e15e0132c2b3c"
server_key_version: "encryptionkey@0"
I20260620 18:36:30.331696 30598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/minicluster-data/ts-2/wal/instance:
uuid: "622c643eaee54366b09419fb11891fe9"
format_stamp: "Formatted at 2026-06-20 18:36:30 on dist-test-slave-5lgp"
server_key: "3399f6cd32653191f7bcee6011606f87"
server_key_iv: "0a9a8d09fb3770c35b0e15e0132c2b3c"
server_key_version: "encryptionkey@0"
I20260620 18:36:30.336755 30598 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 18:36:30.340380 30613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:30.342013 30598 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:36:30.342221 30598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/minicluster-data/ts-2/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/minicluster-data/ts-2/wal
uuid: "622c643eaee54366b09419fb11891fe9"
format_stamp: "Formatted at 2026-06-20 18:36:30 on dist-test-slave-5lgp"
server_key: "3399f6cd32653191f7bcee6011606f87"
server_key_iv: "0a9a8d09fb3770c35b0e15e0132c2b3c"
server_key_version: "encryptionkey@0"
I20260620 18:36:30.342459 30598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:36:30.362636 30598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:36:30.367168 30598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:36:30.367466 30598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:36:30.370302 30598 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:36:30.370419 30598 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:30.370695 30598 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:36:30.370834 30598 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 18:36:30 dist-test-slave-5lgp krb5kdc[30203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980590, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.37.3@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM
I20260620 18:36:30.395143 30343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:30.370930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.3:32929 (local address 127.22.37.62:42963)
0620 18:36:30.371131 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:30.371137 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:30.372471 (+ 1334us) server_negotiation.cc:407] Connection header received
0620 18:36:30.374590 (+ 2119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:30.374596 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:30.374692 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:30.374829 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:30.377535 (+ 2706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:30.378875 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:30.381171 (+ 2296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:30.381894 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:30.382069 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:30.387142 (+ 5073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:30.387147 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:30.387151 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:30.387183 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:30.390736 (+ 3553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:30.391830 (+ 1094us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:30.391836 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:30.391840 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:30.391999 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:30.392514 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:30.392520 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:30.392523 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:30.392897 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:30.393064 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:30.394258 (+ 1194us) server_negotiation.cc:299] Negotiation successful
0620 18:36:30.394594 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 18:36:30.396247 30622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:30.371380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:42963 (local address 127.22.37.3:32929)
0620 18:36:30.372261 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:30.372316 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:30.374290 (+ 1974us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:30.375134 (+ 844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:30.375161 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:30.376326 (+ 1165us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:30.377282 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:30.377306 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:30.379064 (+ 1758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:30.379072 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:30.381008 (+ 1936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:30.381020 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:30.382190 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:30.382202 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:30.382508 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:30.383391 (+ 883us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:30.383418 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:30.386893 (+ 3475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:36:30.391040 (+ 4147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:30.391049 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:30.391065 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:30.391668 (+ 603us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:30.392127 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:30.392131 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:30.392134 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:30.392381 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:30.393132 (+ 751us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:36:30.393140 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:36:30.393658 (+ 518us) client_negotiation.cc:769] Sending connection context
0620 18:36:30.393989 (+ 331us) client_negotiation.cc:240] Negotiation successful
0620 18:36:30.394788 (+ 799us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":375,"threads_started":1}
I20260620 18:36:30.399236 30455 heartbeater.cc:503] Master 127.22.37.62:42963 was elected leader, sending a full tablet report...
I20260620 18:36:30.432542 30343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:30.416254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.3:35979 (local address 127.22.37.62:42963)
0620 18:36:30.416499 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:30.416509 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:30.417147 (+ 638us) server_negotiation.cc:407] Connection header received
0620 18:36:30.417360 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:30.417364 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:30.417517 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:30.417639 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:30.419159 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:30.419924 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:30.422485 (+ 2561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:30.423096 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:30.423233 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:30.425189 (+ 1956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:30.425201 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:30.425204 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:30.425237 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:30.429585 (+ 4348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:30.430332 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:30.430341 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:30.430348 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:30.430523 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:30.431022 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:30.431026 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:30.431028 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:30.431329 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:30.431488 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:30.431933 (+ 445us) server_negotiation.cc:299] Negotiation successful
0620 18:36:30.432158 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 18:36:30.432677 30711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:30.416222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:42963 (local address 127.22.37.3:35979)
0620 18:36:30.416996 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:30.417037 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:30.417229 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:30.417694 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:30.417702 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:30.418498 (+ 796us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:30.418990 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:30.419006 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:30.420079 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:30.420087 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:30.422247 (+ 2160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:30.422281 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:30.423304 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:30.423310 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:30.423464 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:30.424258 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:30.424275 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:30.425000 (+ 725us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:36:30.429750 (+ 4750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:30.429757 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:30.429767 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:30.430198 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:30.430659 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:30.430666 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:30.430672 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:30.430856 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:30.431452 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:36:30.431458 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:36:30.431669 (+ 211us) client_negotiation.cc:769] Sending connection context
0620 18:36:30.431820 (+ 151us) client_negotiation.cc:240] Negotiation successful
0620 18:36:30.432213 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":390,"threads_started":1}
I20260620 18:36:30.434495 30598 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.3:38211
I20260620 18:36:30.434520 30732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.3:38211 every 8 connection(s)
I20260620 18:36:30.435855 30598 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/minicluster-data/ts-2/data/info.pb
I20260620 18:36:30.437901 30733 heartbeater.cc:347] Connected to a master server at 127.22.37.62:42963
I20260620 18:36:30.438130 30733 heartbeater.cc:464] Registering TS with master...
I20260620 18:36:30.438760 30733 heartbeater.cc:511] Master 127.22.37.62:42963 requested a full tablet report, sending...
I20260620 18:36:30.440044 30248 ts_manager.cc:194] Registered new tserver with Master: 622c643eaee54366b09419fb11891fe9 (127.22.37.3:38211)
I20260620 18:36:30.441001 30248 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.37.3@KRBTEST.COM'} at 127.22.37.3:32929
I20260620 18:36:30.442102 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 30598
I20260620 18:36:30.442304 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/minicluster-data/ts-2/wal/instance
I20260620 18:36:30.442839 22676 external_mini_cluster.cc:1428] Setting key 19b3dce7184f1bbbdd96c44a3b4a45ad
I20260620 18:36:30.445755 22676 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 18:36:30 dist-test-slave-5lgp krb5kdc[30203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980590, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 18:36:30 dist-test-slave-5lgp krb5kdc[30203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:36:30 dist-test-slave-5lgp krb5kdc[30203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 18:36:30.496757 30343 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:36:30.486157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43572 (local address 127.22.37.62:42963)
0620 18:36:30.486419 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:30.486427 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:30.486559 (+ 132us) server_negotiation.cc:407] Connection header received
0620 18:36:30.486691 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:30.486695 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:30.486760 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:30.486849 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:30.488257 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:30.488985 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:30.490308 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:30.490745 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:30.490850 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:30.496520 (+ 5670us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43572: BlockingRecv error: recv got EOF from 127.0.0.1:43572 (error 108)
Metrics: {"server-negotiator.queue_time_us":110}
Jun 20 18:36:30 dist-test-slave-5lgp krb5kdc[30203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980590, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM
I20260620 18:36:30.514199 30343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:30.500068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43586 (local address 127.22.37.62:42963)
0620 18:36:30.500283 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:30.500291 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:30.500646 (+ 355us) server_negotiation.cc:407] Connection header received
0620 18:36:30.500789 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:30.500793 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:30.500869 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:30.500954 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:30.502177 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:30.502923 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:30.504248 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:30.504655 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:30.504813 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:30.508771 (+ 3958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:30.508776 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:30.508780 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:30.508814 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:30.511513 (+ 2699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:30.512193 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:30.512200 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:30.512208 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:30.512331 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:30.512708 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:30.512714 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:30.512717 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:30.513027 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:30.513338 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:30.513658 (+ 320us) server_negotiation.cc:299] Negotiation successful
0620 18:36:30.513887 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 18:36:30.518389 30248 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43586:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:36:30.525607 30248 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:36:30.549194 30752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:30.538842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44078 (local address 127.22.37.3:38211)
0620 18:36:30.539583 (+ 741us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:30.539594 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:30.540078 (+ 484us) server_negotiation.cc:407] Connection header received
0620 18:36:30.540391 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:30.540400 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:30.540671 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:30.540915 (+ 244us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:36:30.541993 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:30.543659 (+ 1666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:30.546586 (+ 2927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:30.547835 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:30.548113 (+ 278us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:30.548383 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:30.548674 (+ 291us) server_negotiation.cc:299] Negotiation successful
0620 18:36:30.548851 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":346,"threads_started":1}
I20260620 18:36:30.549301 30751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:30.538655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.3:38211 (local address 127.0.0.1:44078)
0620 18:36:30.539926 (+ 1271us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:30.539962 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:30.540172 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:30.541324 (+ 1152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:30.541332 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:30.541407 (+ 75us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:36:30.541777 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:30.541790 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:30.543777 (+ 1987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:30.543782 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:30.546378 (+ 2596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:30.546395 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:30.548181 (+ 1786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:30.548188 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:30.548429 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:30.548476 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 18:36:30.548616 (+ 140us) client_negotiation.cc:240] Negotiation successful
0620 18:36:30.548827 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1083,"mutex_wait_us":1,"thread_start_us":930,"threads_started":1}
I20260620 18:36:30.550132 30753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:30.540300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.2:46153 (local address 127.0.0.1:48596)
0620 18:36:30.540949 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:30.540972 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:30.541078 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:30.542339 (+ 1261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:30.542346 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:30.542381 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:36:30.542929 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:30.542941 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:30.544796 (+ 1855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:30.544801 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:30.546940 (+ 2139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:30.546953 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:30.548921 (+ 1968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:30.548934 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:30.549092 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:30.549124 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 18:36:30.549189 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 18:36:30.549746 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"mutex_wait_us":373,"thread_start_us":380,"threads_started":1}
I20260620 18:36:30.550628 30754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:30.540574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48596 (local address 127.22.37.2:46153)
0620 18:36:30.541582 (+ 1008us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:30.541591 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:30.541676 (+ 85us) server_negotiation.cc:407] Connection header received
0620 18:36:30.541778 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:30.541788 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:30.542080 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:30.543245 (+ 1165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:36:30.543430 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:30.544647 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:30.547158 (+ 2511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:30.548676 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:30.549423 (+ 747us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:30.550042 (+ 619us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:30.550127 (+ 85us) server_negotiation.cc:299] Negotiation successful
0620 18:36:30.550338 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":849,"thread_start_us":695,"threads_started":1}
I20260620 18:36:30.553349 30663 tablet_service.cc:1467] Processing CreateTablet for tablet 4ac89d458ebe4aff8428dcf0b2db00a6 (DEFAULT_TABLE table=test-table [id=1a470a0964da49899bd339bdd785ff73]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:36:30.553498 30524 tablet_service.cc:1467] Processing CreateTablet for tablet 4ac89d458ebe4aff8428dcf0b2db00a6 (DEFAULT_TABLE table=test-table [id=1a470a0964da49899bd339bdd785ff73]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:36:30.554382 30755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:30.541477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55434 (local address 127.22.37.1:44231)
0620 18:36:30.543110 (+ 1633us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:30.543119 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:30.543508 (+ 389us) server_negotiation.cc:407] Connection header received
0620 18:36:30.543777 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:30.543785 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:30.544050 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:30.544241 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:36:30.544910 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:30.547005 (+ 2095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:30.551818 (+ 4813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:30.553280 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:30.553458 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:30.553672 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:30.553927 (+ 255us) server_negotiation.cc:299] Negotiation successful
0620 18:36:30.554088 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1481,"thread_start_us":1382,"threads_started":1}
I20260620 18:36:30.554672 30756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:30.541646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.1:44231 (local address 127.0.0.1:55434)
0620 18:36:30.543392 (+ 1746us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:30.543410 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:30.543618 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:30.544261 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:30.544266 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:30.544291 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:36:30.544736 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:30.544750 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:30.547222 (+ 2472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:30.547227 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:30.550657 (+ 3430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:30.550673 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:30.553533 (+ 2860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:30.553540 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:30.553724 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:30.553770 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 18:36:30.553869 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 18:36:30.554008 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1637,"mutex_wait_us":1,"thread_start_us":1510,"threads_started":1}
I20260620 18:36:30.554952 30663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ac89d458ebe4aff8428dcf0b2db00a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:36:30.554958 30524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ac89d458ebe4aff8428dcf0b2db00a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:36:30.558007 30385 tablet_service.cc:1467] Processing CreateTablet for tablet 4ac89d458ebe4aff8428dcf0b2db00a6 (DEFAULT_TABLE table=test-table [id=1a470a0964da49899bd339bdd785ff73]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:36:30.559430 30385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ac89d458ebe4aff8428dcf0b2db00a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:36:30.566119 30757 tablet_bootstrap.cc:492] T 4ac89d458ebe4aff8428dcf0b2db00a6 P 856016baca2c4cf3b35c8b1c3d8ac989: Bootstrap starting.
I20260620 18:36:30.569995 30757 tablet_bootstrap.cc:654] T 4ac89d458ebe4aff8428dcf0b2db00a6 P 856016baca2c4cf3b35c8b1c3d8ac989: Neither blocks nor log segments found. Creating new log.
I20260620 18:36:30.571295 30757 log.cc:824] T 4ac89d458ebe4aff8428dcf0b2db00a6 P 856016baca2c4cf3b35c8b1c3d8ac989: Log is configured to *not* fsync() on all Append() calls
I20260620 18:36:30.571441 30758 tablet_bootstrap.cc:492] T 4ac89d458ebe4aff8428dcf0b2db00a6 P 622c643eaee54366b09419fb11891fe9: Bootstrap starting.
I20260620 18:36:30.574251 30757 tablet_bootstrap.cc:492] T 4ac89d458ebe4aff8428dcf0b2db00a6 P 856016baca2c4cf3b35c8b1c3d8ac989: No bootstrap required, opened a new log
I20260620 18:36:30.574548 30757 ts_tablet_manager.cc:1397] T 4ac89d458ebe4aff8428dcf0b2db00a6 P 856016baca2c4cf3b35c8b1c3d8ac989: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260620 18:36:30.574707 30760 tablet_bootstrap.cc:492] T 4ac89d458ebe4aff8428dcf0b2db00a6 P b94701dfaaea4c6c95f229005e7b613e: Bootstrap starting.
I20260620 18:36:30.575282 30758 tablet_bootstrap.cc:654] T 4ac89d458ebe4aff8428dcf0b2db00a6 P 622c643eaee54366b09419fb11891fe9: Neither blocks nor log segments found. Creating new log.
I20260620 18:36:30.576386 30758 log.cc:824] T 4ac89d458ebe4aff8428dcf0b2db00a6 P 622c643eaee54366b09419fb11891fe9: Log is configured to *not* fsync() on all Append() calls
I20260620 18:36:30.578510 30760 tablet_bootstrap.cc:654] T 4ac89d458ebe4aff8428dcf0b2db00a6 P b94701dfaaea4c6c95f229005e7b613e: Neither blocks nor log segments found. Creating new log.
I20260620 18:36:30.579757 30760 log.cc:824] T 4ac89d458ebe4aff8428dcf0b2db00a6 P b94701dfaaea4c6c95f229005e7b613e: Log is configured to *not* fsync() on all Append() calls
I20260620 18:36:30.581768 30758 tablet_bootstrap.cc:492] T 4ac89d458ebe4aff8428dcf0b2db00a6 P 622c643eaee54366b09419fb11891fe9: No bootstrap required, opened a new log
I20260620 18:36:30.582048 30758 ts_tablet_manager.cc:1397] T 4ac89d458ebe4aff8428dcf0b2db00a6 P 622c643eaee54366b09419fb11891fe9: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20260620 18:36:30.582748 30760 tablet_bootstrap.cc:492] T 4ac89d458ebe4aff8428dcf0b2db00a6 P b94701dfaaea4c6c95f229005e7b613e: No bootstrap required, opened a new log
I20260620 18:36:30.583009 30760 ts_tablet_manager.cc:1397] T 4ac89d458ebe4aff8428dcf0b2db00a6 P b94701dfaaea4c6c95f229005e7b613e: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260620 18:36:30.584956 30757 raft_consensus.cc:348] T 4ac89d458ebe4aff8428dcf0b2db00a6 P 856016baca2c4cf3b35c8b1c3d8ac989 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "622c643eaee54366b09419fb11891fe9" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 38211 } } peers { permanent_uuid: "856016baca2c4cf3b35c8b1c3d8ac989" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 46153 } } peers { permanent_uuid: "b94701dfaaea4c6c95f229005e7b613e" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44231 } }
I20260620 18:36:30.585434 30757 raft_consensus.cc:374] T 4ac89d458ebe4aff8428dcf0b2db00a6 P 856016baca2c4cf3b35c8b1c3d8ac989 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:36:30.585533 30757 raft_consensus.cc:729] T 4ac89d458ebe4aff8428dcf0b2db00a6 P 856016baca2c4cf3b35c8b1c3d8ac989 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 856016baca2c4cf3b35c8b1c3d8ac989, State: Initialized, Role: FOLLOWER
I20260620 18:36:30.586180 30757 consensus_queue.cc:260] T 4ac89d458ebe4aff8428dcf0b2db00a6 P 856016baca2c4cf3b35c8b1c3d8ac989 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "622c643eaee54366b09419fb11891fe9" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 38211 } } peers { permanent_uuid: "856016baca2c4cf3b35c8b1c3d8ac989" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 46153 } } peers { permanent_uuid: "b94701dfaaea4c6c95f229005e7b613e" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44231 } }
I20260620 18:36:30.587267 30594 heartbeater.cc:503] Master 127.22.37.62:42963 was elected leader, sending a full tablet report...
I20260620 18:36:30.590257 30757 ts_tablet_manager.cc:1428] T 4ac89d458ebe4aff8428dcf0b2db00a6 P 856016baca2c4cf3b35c8b1c3d8ac989: Time spent starting tablet: real 0.016s user 0.008s sys 0.006s
I20260620 18:36:30.590809 30758 raft_consensus.cc:348] T 4ac89d458ebe4aff8428dcf0b2db00a6 P 622c643eaee54366b09419fb11891fe9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "622c643eaee54366b09419fb11891fe9" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 38211 } } peers { permanent_uuid: "856016baca2c4cf3b35c8b1c3d8ac989" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 46153 } } peers { permanent_uuid: "b94701dfaaea4c6c95f229005e7b613e" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44231 } }
I20260620 18:36:30.591264 30758 raft_consensus.cc:374] T 4ac89d458ebe4aff8428dcf0b2db00a6 P 622c643eaee54366b09419fb11891fe9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:36:30.591382 30758 raft_consensus.cc:729] T 4ac89d458ebe4aff8428dcf0b2db00a6 P 622c643eaee54366b09419fb11891fe9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 622c643eaee54366b09419fb11891fe9, State: Initialized, Role: FOLLOWER
I20260620 18:36:30.592033 30758 consensus_queue.cc:260] T 4ac89d458ebe4aff8428dcf0b2db00a6 P 622c643eaee54366b09419fb11891fe9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "622c643eaee54366b09419fb11891fe9" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 38211 } } peers { permanent_uuid: "856016baca2c4cf3b35c8b1c3d8ac989" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 46153 } } peers { permanent_uuid: "b94701dfaaea4c6c95f229005e7b613e" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44231 } }
I20260620 18:36:30.593143 30733 heartbeater.cc:503] Master 127.22.37.62:42963 was elected leader, sending a full tablet report...
I20260620 18:36:30.593046 30760 raft_consensus.cc:348] T 4ac89d458ebe4aff8428dcf0b2db00a6 P b94701dfaaea4c6c95f229005e7b613e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "622c643eaee54366b09419fb11891fe9" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 38211 } } peers { permanent_uuid: "856016baca2c4cf3b35c8b1c3d8ac989" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 46153 } } peers { permanent_uuid: "b94701dfaaea4c6c95f229005e7b613e" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44231 } }
I20260620 18:36:30.593505 30760 raft_consensus.cc:374] T 4ac89d458ebe4aff8428dcf0b2db00a6 P b94701dfaaea4c6c95f229005e7b613e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:36:30.593604 30760 raft_consensus.cc:729] T 4ac89d458ebe4aff8428dcf0b2db00a6 P b94701dfaaea4c6c95f229005e7b613e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b94701dfaaea4c6c95f229005e7b613e, State: Initialized, Role: FOLLOWER
I20260620 18:36:30.593886 30758 ts_tablet_manager.cc:1428] T 4ac89d458ebe4aff8428dcf0b2db00a6 P 622c643eaee54366b09419fb11891fe9: Time spent starting tablet: real 0.012s user 0.011s sys 0.001s
I20260620 18:36:30.594307 30760 consensus_queue.cc:260] T 4ac89d458ebe4aff8428dcf0b2db00a6 P b94701dfaaea4c6c95f229005e7b613e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "622c643eaee54366b09419fb11891fe9" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 38211 } } peers { permanent_uuid: "856016baca2c4cf3b35c8b1c3d8ac989" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 46153 } } peers { permanent_uuid: "b94701dfaaea4c6c95f229005e7b613e" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44231 } }
I20260620 18:36:30.596118 30760 ts_tablet_manager.cc:1428] T 4ac89d458ebe4aff8428dcf0b2db00a6 P b94701dfaaea4c6c95f229005e7b613e: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
W20260620 18:36:30.642976 30456 tablet.cc:2369] T 4ac89d458ebe4aff8428dcf0b2db00a6 P b94701dfaaea4c6c95f229005e7b613e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:36:30.662472 30595 tablet.cc:2369] T 4ac89d458ebe4aff8428dcf0b2db00a6 P 856016baca2c4cf3b35c8b1c3d8ac989: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:36:30.687546 30734 tablet.cc:2369] T 4ac89d458ebe4aff8428dcf0b2db00a6 P 622c643eaee54366b09419fb11891fe9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:36:31.008634 30765 raft_consensus.cc:482] T 4ac89d458ebe4aff8428dcf0b2db00a6 P b94701dfaaea4c6c95f229005e7b613e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:36:31.008991 30765 raft_consensus.cc:504] T 4ac89d458ebe4aff8428dcf0b2db00a6 P b94701dfaaea4c6c95f229005e7b613e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "622c643eaee54366b09419fb11891fe9" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 38211 } } peers { permanent_uuid: "856016baca2c4cf3b35c8b1c3d8ac989" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 46153 } } peers { permanent_uuid: "b94701dfaaea4c6c95f229005e7b613e" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44231 } }
I20260620 18:36:31.010650 30765 leader_election.cc:288] T 4ac89d458ebe4aff8428dcf0b2db00a6 P b94701dfaaea4c6c95f229005e7b613e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 622c643eaee54366b09419fb11891fe9 (127.22.37.3:38211), 856016baca2c4cf3b35c8b1c3d8ac989 (127.22.37.2:46153)
I20260620 18:36:31.019014 30752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:31.011055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.1:38499 (local address 127.22.37.3:38211)
0620 18:36:31.011251 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:31.011256 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:31.011774 (+ 518us) server_negotiation.cc:407] Connection header received
0620 18:36:31.011962 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:31.011970 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:31.012051 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:31.012165 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:36:31.012923 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:31.013965 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:31.016450 (+ 2485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:31.017681 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:31.017819 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:31.017962 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:31.018661 (+ 699us) server_negotiation.cc:299] Negotiation successful
0620 18:36:31.018788 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260620 18:36:31.019014 30754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:31.012057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.1:36929 (local address 127.22.37.2:46153)
0620 18:36:31.012277 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:31.012284 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:31.012461 (+ 177us) server_negotiation.cc:407] Connection header received
0620 18:36:31.012647 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:31.012650 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:31.012705 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:31.012776 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:36:31.013273 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:31.014323 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:31.016633 (+ 2310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:31.017829 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:31.017949 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:31.018046 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:31.018637 (+ 591us) server_negotiation.cc:299] Negotiation successful
0620 18:36:31.018760 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260620 18:36:31.019277 30766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:31.010988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.3:38211 (local address 127.22.37.1:38499)
0620 18:36:31.011643 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:31.011664 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:31.011807 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:31.012314 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:31.012319 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:31.012346 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:36:31.012744 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:31.012778 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:31.014088 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:31.014092 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:31.016296 (+ 2204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:31.016310 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:31.018110 (+ 1800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:31.018116 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:31.018414 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:31.018477 (+ 63us) client_negotiation.cc:769] Sending connection context
0620 18:36:31.018620 (+ 143us) client_negotiation.cc:240] Negotiation successful
0620 18:36:31.018720 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"mutex_wait_us":64,"thread_start_us":309,"threads_started":1}
I20260620 18:36:31.019376 30767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:31.011761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.2:46153 (local address 127.22.37.1:36929)
0620 18:36:31.012363 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:31.012380 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:31.012516 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:31.012813 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:31.012820 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:31.012850 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:36:31.013120 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:31.013126 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:31.014471 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:31.014476 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:31.016495 (+ 2019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:31.016504 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:31.018109 (+ 1605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:31.018116 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:31.018416 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:31.018480 (+ 64us) client_negotiation.cc:769] Sending connection context
0620 18:36:31.018581 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 18:36:31.018713 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"mutex_wait_us":89,"thread_start_us":335,"threads_started":1}
I20260620 18:36:31.020128 30685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ac89d458ebe4aff8428dcf0b2db00a6" candidate_uuid: "b94701dfaaea4c6c95f229005e7b613e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "622c643eaee54366b09419fb11891fe9" is_pre_election: true
I20260620 18:36:31.020277 30549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ac89d458ebe4aff8428dcf0b2db00a6" candidate_uuid: "b94701dfaaea4c6c95f229005e7b613e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "856016baca2c4cf3b35c8b1c3d8ac989" is_pre_election: true
I20260620 18:36:31.020609 30685 raft_consensus.cc:2454] T 4ac89d458ebe4aff8428dcf0b2db00a6 P 622c643eaee54366b09419fb11891fe9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b94701dfaaea4c6c95f229005e7b613e in term 0.
I20260620 18:36:31.020794 30549 raft_consensus.cc:2454] T 4ac89d458ebe4aff8428dcf0b2db00a6 P 856016baca2c4cf3b35c8b1c3d8ac989 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b94701dfaaea4c6c95f229005e7b613e in term 0.
I20260620 18:36:31.021206 30338 leader_election.cc:302] T 4ac89d458ebe4aff8428dcf0b2db00a6 P b94701dfaaea4c6c95f229005e7b613e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 622c643eaee54366b09419fb11891fe9, b94701dfaaea4c6c95f229005e7b613e; no voters:
I20260620 18:36:31.021612 30765 raft_consensus.cc:2781] T 4ac89d458ebe4aff8428dcf0b2db00a6 P b94701dfaaea4c6c95f229005e7b613e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:36:31.021770 30765 raft_consensus.cc:482] T 4ac89d458ebe4aff8428dcf0b2db00a6 P b94701dfaaea4c6c95f229005e7b613e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:36:31.021874 30765 raft_consensus.cc:3037] T 4ac89d458ebe4aff8428dcf0b2db00a6 P b94701dfaaea4c6c95f229005e7b613e [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:36:31.024250 30765 raft_consensus.cc:504] T 4ac89d458ebe4aff8428dcf0b2db00a6 P b94701dfaaea4c6c95f229005e7b613e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "622c643eaee54366b09419fb11891fe9" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 38211 } } peers { permanent_uuid: "856016baca2c4cf3b35c8b1c3d8ac989" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 46153 } } peers { permanent_uuid: "b94701dfaaea4c6c95f229005e7b613e" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44231 } }
I20260620 18:36:31.024811 30765 leader_election.cc:288] T 4ac89d458ebe4aff8428dcf0b2db00a6 P b94701dfaaea4c6c95f229005e7b613e [CANDIDATE]: Term 1 election: Requested vote from peers 622c643eaee54366b09419fb11891fe9 (127.22.37.3:38211), 856016baca2c4cf3b35c8b1c3d8ac989 (127.22.37.2:46153)
I20260620 18:36:31.025249 30685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ac89d458ebe4aff8428dcf0b2db00a6" candidate_uuid: "b94701dfaaea4c6c95f229005e7b613e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "622c643eaee54366b09419fb11891fe9"
I20260620 18:36:31.025290 30549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ac89d458ebe4aff8428dcf0b2db00a6" candidate_uuid: "b94701dfaaea4c6c95f229005e7b613e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "856016baca2c4cf3b35c8b1c3d8ac989"
I20260620 18:36:31.025465 30685 raft_consensus.cc:3037] T 4ac89d458ebe4aff8428dcf0b2db00a6 P 622c643eaee54366b09419fb11891fe9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:36:31.025520 30549 raft_consensus.cc:3037] T 4ac89d458ebe4aff8428dcf0b2db00a6 P 856016baca2c4cf3b35c8b1c3d8ac989 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:36:31.027737 30549 raft_consensus.cc:2454] T 4ac89d458ebe4aff8428dcf0b2db00a6 P 856016baca2c4cf3b35c8b1c3d8ac989 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b94701dfaaea4c6c95f229005e7b613e in term 1.
I20260620 18:36:31.027747 30685 raft_consensus.cc:2454] T 4ac89d458ebe4aff8428dcf0b2db00a6 P 622c643eaee54366b09419fb11891fe9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b94701dfaaea4c6c95f229005e7b613e in term 1.
I20260620 18:36:31.028123 30338 leader_election.cc:302] T 4ac89d458ebe4aff8428dcf0b2db00a6 P b94701dfaaea4c6c95f229005e7b613e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 856016baca2c4cf3b35c8b1c3d8ac989, b94701dfaaea4c6c95f229005e7b613e; no voters:
I20260620 18:36:31.028436 30765 raft_consensus.cc:2781] T 4ac89d458ebe4aff8428dcf0b2db00a6 P b94701dfaaea4c6c95f229005e7b613e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:36:31.028846 30765 raft_consensus.cc:686] T 4ac89d458ebe4aff8428dcf0b2db00a6 P b94701dfaaea4c6c95f229005e7b613e [term 1 LEADER]: Becoming Leader. State: Replica: b94701dfaaea4c6c95f229005e7b613e, State: Running, Role: LEADER
I20260620 18:36:31.029614 30765 consensus_queue.cc:237] T 4ac89d458ebe4aff8428dcf0b2db00a6 P b94701dfaaea4c6c95f229005e7b613e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "622c643eaee54366b09419fb11891fe9" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 38211 } } peers { permanent_uuid: "856016baca2c4cf3b35c8b1c3d8ac989" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 46153 } } peers { permanent_uuid: "b94701dfaaea4c6c95f229005e7b613e" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44231 } }
I20260620 18:36:31.034005 30247 catalog_manager.cc:5495] T 4ac89d458ebe4aff8428dcf0b2db00a6 P b94701dfaaea4c6c95f229005e7b613e reported cstate change: term changed from 0 to 1, leader changed from <none> to b94701dfaaea4c6c95f229005e7b613e (127.22.37.1). New cstate: current_term: 1 leader_uuid: "b94701dfaaea4c6c95f229005e7b613e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "622c643eaee54366b09419fb11891fe9" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 38211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "856016baca2c4cf3b35c8b1c3d8ac989" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 46153 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b94701dfaaea4c6c95f229005e7b613e" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44231 } health_report { overall_health: HEALTHY } } }
I20260620 18:36:31.115707 30774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:31.109878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55442 (local address 127.22.37.1:44231)
0620 18:36:31.110301 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:31.110306 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:31.110528 (+ 222us) server_negotiation.cc:407] Connection header received
0620 18:36:31.110724 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:31.110728 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:31.110808 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:31.110939 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:36:31.111407 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:31.112061 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:31.113710 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:31.114101 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:31.114196 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:31.114572 (+ 376us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:36:31.115116 (+ 544us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:36:31.115233 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:31.115411 (+ 178us) server_negotiation.cc:299] Negotiation successful
0620 18:36:31.115482 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":240,"threads_started":1}
I20260620 18:36:31.123502 30549 raft_consensus.cc:1261] T 4ac89d458ebe4aff8428dcf0b2db00a6 P 856016baca2c4cf3b35c8b1c3d8ac989 [term 1 FOLLOWER]: Refusing update from remote peer b94701dfaaea4c6c95f229005e7b613e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:36:31.123643 30685 raft_consensus.cc:1261] T 4ac89d458ebe4aff8428dcf0b2db00a6 P 622c643eaee54366b09419fb11891fe9 [term 1 FOLLOWER]: Refusing update from remote peer b94701dfaaea4c6c95f229005e7b613e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:36:31.124483 30768 consensus_queue.cc:1035] T 4ac89d458ebe4aff8428dcf0b2db00a6 P b94701dfaaea4c6c95f229005e7b613e [LEADER]: Connected to new peer: Peer: permanent_uuid: "856016baca2c4cf3b35c8b1c3d8ac989" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 46153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:36:31.124809 30765 consensus_queue.cc:1035] T 4ac89d458ebe4aff8428dcf0b2db00a6 P b94701dfaaea4c6c95f229005e7b613e [LEADER]: Connected to new peer: Peer: permanent_uuid: "622c643eaee54366b09419fb11891fe9" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 38211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:36:31.141904 30780 mvcc.cc:205] Tried to move back new op lower bound from 7298992501231939584 to 7298992500860837888. Current Snapshot: MvccSnapshot[applied={T|T < 7298992501231939584}]
I20260620 18:36:31.154192 30248 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43586:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:36:31.154412 30248 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43586:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:36:31.157727 30248 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4d9466ac6e3e4d1c95493d9373c03f45: Sending DeleteTablet for 3 replicas of tablet 4ac89d458ebe4aff8428dcf0b2db00a6
I20260620 18:36:31.158828 30663 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ac89d458ebe4aff8428dcf0b2db00a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:36:31 UTC) from {username='oryx', principal='oryx/127.22.37.62@KRBTEST.COM'} at 127.0.0.1:44078
I20260620 18:36:31.159207 30385 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ac89d458ebe4aff8428dcf0b2db00a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:36:31 UTC) from {username='oryx', principal='oryx/127.22.37.62@KRBTEST.COM'} at 127.0.0.1:55434
I20260620 18:36:31.159533 30524 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ac89d458ebe4aff8428dcf0b2db00a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:36:31 UTC) from {username='oryx', principal='oryx/127.22.37.62@KRBTEST.COM'} at 127.0.0.1:48596
I20260620 18:36:31.159906 30248 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43586:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:36:31.160661 30248 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:36:31.163151 30789 tablet_replica.cc:330] T 4ac89d458ebe4aff8428dcf0b2db00a6 P 856016baca2c4cf3b35c8b1c3d8ac989: stopping tablet replica
I20260620 18:36:31.163592 30789 raft_consensus.cc:2229] T 4ac89d458ebe4aff8428dcf0b2db00a6 P 856016baca2c4cf3b35c8b1c3d8ac989 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:36:31.164120 30789 raft_consensus.cc:2258] T 4ac89d458ebe4aff8428dcf0b2db00a6 P 856016baca2c4cf3b35c8b1c3d8ac989 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:36:31.164904 30787 tablet_replica.cc:330] T 4ac89d458ebe4aff8428dcf0b2db00a6 P 622c643eaee54366b09419fb11891fe9: stopping tablet replica
I20260620 18:36:31.165319 30787 raft_consensus.cc:2229] T 4ac89d458ebe4aff8428dcf0b2db00a6 P 622c643eaee54366b09419fb11891fe9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:36:31.165792 30787 raft_consensus.cc:2258] T 4ac89d458ebe4aff8428dcf0b2db00a6 P 622c643eaee54366b09419fb11891fe9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:36:31.166443 30789 ts_tablet_manager.cc:1905] T 4ac89d458ebe4aff8428dcf0b2db00a6 P 856016baca2c4cf3b35c8b1c3d8ac989: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:36:31.166512 30788 tablet_replica.cc:330] T 4ac89d458ebe4aff8428dcf0b2db00a6 P b94701dfaaea4c6c95f229005e7b613e: stopping tablet replica
I20260620 18:36:31.166884 30788 raft_consensus.cc:2229] T 4ac89d458ebe4aff8428dcf0b2db00a6 P b94701dfaaea4c6c95f229005e7b613e [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:36:31.167572 30788 raft_consensus.cc:2258] T 4ac89d458ebe4aff8428dcf0b2db00a6 P b94701dfaaea4c6c95f229005e7b613e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:36:31.169497 30787 ts_tablet_manager.cc:1905] T 4ac89d458ebe4aff8428dcf0b2db00a6 P 622c643eaee54366b09419fb11891fe9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:36:31.170471 30662 tablet_service.cc:1467] Processing CreateTablet for tablet 3bdabbd822b34f04a0f188f544221a94 (DEFAULT_TABLE table=test-table [id=032b0a3a56f44eba9e72e979df60ec00]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:36:31.171003 30662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bdabbd822b34f04a0f188f544221a94. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:36:31.173475 30789 ts_tablet_manager.cc:1918] T 4ac89d458ebe4aff8428dcf0b2db00a6 P 856016baca2c4cf3b35c8b1c3d8ac989: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:36:31.173641 30789 log.cc:1196] T 4ac89d458ebe4aff8428dcf0b2db00a6 P 856016baca2c4cf3b35c8b1c3d8ac989: Deleting WAL directory at /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/minicluster-data/ts-1/wal/wals/4ac89d458ebe4aff8428dcf0b2db00a6
I20260620 18:36:31.174181 30789 ts_tablet_manager.cc:1939] T 4ac89d458ebe4aff8428dcf0b2db00a6 P 856016baca2c4cf3b35c8b1c3d8ac989: Deleting consensus metadata
I20260620 18:36:31.174104 30524 tablet_service.cc:1467] Processing CreateTablet for tablet 3bdabbd822b34f04a0f188f544221a94 (DEFAULT_TABLE table=test-table [id=032b0a3a56f44eba9e72e979df60ec00]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:36:31.174558 30524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bdabbd822b34f04a0f188f544221a94. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:36:31.176329 30787 ts_tablet_manager.cc:1918] T 4ac89d458ebe4aff8428dcf0b2db00a6 P 622c643eaee54366b09419fb11891fe9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:36:31.176467 30787 log.cc:1196] T 4ac89d458ebe4aff8428dcf0b2db00a6 P 622c643eaee54366b09419fb11891fe9: Deleting WAL directory at /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/minicluster-data/ts-2/wal/wals/4ac89d458ebe4aff8428dcf0b2db00a6
I20260620 18:36:31.176976 30787 ts_tablet_manager.cc:1939] T 4ac89d458ebe4aff8428dcf0b2db00a6 P 622c643eaee54366b09419fb11891fe9: Deleting consensus metadata
I20260620 18:36:31.178310 30385 tablet_service.cc:1467] Processing CreateTablet for tablet 3bdabbd822b34f04a0f188f544221a94 (DEFAULT_TABLE table=test-table [id=032b0a3a56f44eba9e72e979df60ec00]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:36:31.178817 30385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bdabbd822b34f04a0f188f544221a94. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:36:31.178819 30788 ts_tablet_manager.cc:1905] T 4ac89d458ebe4aff8428dcf0b2db00a6 P b94701dfaaea4c6c95f229005e7b613e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:36:31.182370 30235 catalog_manager.cc:4841] TS 856016baca2c4cf3b35c8b1c3d8ac989 (127.22.37.2:46153): tablet 4ac89d458ebe4aff8428dcf0b2db00a6 (table test-table [id=1a470a0964da49899bd339bdd785ff73]) successfully deleted
I20260620 18:36:31.183177 30235 catalog_manager.cc:4841] TS 622c643eaee54366b09419fb11891fe9 (127.22.37.3:38211): tablet 4ac89d458ebe4aff8428dcf0b2db00a6 (table test-table [id=1a470a0964da49899bd339bdd785ff73]) successfully deleted
I20260620 18:36:31.188611 30788 ts_tablet_manager.cc:1918] T 4ac89d458ebe4aff8428dcf0b2db00a6 P b94701dfaaea4c6c95f229005e7b613e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:36:31.188786 30788 log.cc:1196] T 4ac89d458ebe4aff8428dcf0b2db00a6 P b94701dfaaea4c6c95f229005e7b613e: Deleting WAL directory at /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/minicluster-data/ts-0/wal/wals/4ac89d458ebe4aff8428dcf0b2db00a6
I20260620 18:36:31.189287 30791 tablet_bootstrap.cc:492] T 3bdabbd822b34f04a0f188f544221a94 P 622c643eaee54366b09419fb11891fe9: Bootstrap starting.
I20260620 18:36:31.189327 30788 ts_tablet_manager.cc:1939] T 4ac89d458ebe4aff8428dcf0b2db00a6 P b94701dfaaea4c6c95f229005e7b613e: Deleting consensus metadata
I20260620 18:36:31.189942 30790 tablet_bootstrap.cc:492] T 3bdabbd822b34f04a0f188f544221a94 P 856016baca2c4cf3b35c8b1c3d8ac989: Bootstrap starting.
I20260620 18:36:31.191027 30235 catalog_manager.cc:4841] TS b94701dfaaea4c6c95f229005e7b613e (127.22.37.1:44231): tablet 4ac89d458ebe4aff8428dcf0b2db00a6 (table test-table [id=1a470a0964da49899bd339bdd785ff73]) successfully deleted
I20260620 18:36:31.191103 30791 tablet_bootstrap.cc:654] T 3bdabbd822b34f04a0f188f544221a94 P 622c643eaee54366b09419fb11891fe9: Neither blocks nor log segments found. Creating new log.
I20260620 18:36:31.192521 30790 tablet_bootstrap.cc:654] T 3bdabbd822b34f04a0f188f544221a94 P 856016baca2c4cf3b35c8b1c3d8ac989: Neither blocks nor log segments found. Creating new log.
I20260620 18:36:31.196990 30791 tablet_bootstrap.cc:492] T 3bdabbd822b34f04a0f188f544221a94 P 622c643eaee54366b09419fb11891fe9: No bootstrap required, opened a new log
I20260620 18:36:31.197198 30791 ts_tablet_manager.cc:1397] T 3bdabbd822b34f04a0f188f544221a94 P 622c643eaee54366b09419fb11891fe9: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20260620 18:36:31.198102 30791 raft_consensus.cc:348] T 3bdabbd822b34f04a0f188f544221a94 P 622c643eaee54366b09419fb11891fe9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b94701dfaaea4c6c95f229005e7b613e" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44231 } } peers { permanent_uuid: "856016baca2c4cf3b35c8b1c3d8ac989" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 46153 } } peers { permanent_uuid: "622c643eaee54366b09419fb11891fe9" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 38211 } }
I20260620 18:36:31.198441 30791 raft_consensus.cc:374] T 3bdabbd822b34f04a0f188f544221a94 P 622c643eaee54366b09419fb11891fe9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:36:31.198557 30791 raft_consensus.cc:729] T 3bdabbd822b34f04a0f188f544221a94 P 622c643eaee54366b09419fb11891fe9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 622c643eaee54366b09419fb11891fe9, State: Initialized, Role: FOLLOWER
I20260620 18:36:31.198913 30791 consensus_queue.cc:260] T 3bdabbd822b34f04a0f188f544221a94 P 622c643eaee54366b09419fb11891fe9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b94701dfaaea4c6c95f229005e7b613e" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44231 } } peers { permanent_uuid: "856016baca2c4cf3b35c8b1c3d8ac989" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 46153 } } peers { permanent_uuid: "622c643eaee54366b09419fb11891fe9" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 38211 } }
I20260620 18:36:31.199292 30790 tablet_bootstrap.cc:492] T 3bdabbd822b34f04a0f188f544221a94 P 856016baca2c4cf3b35c8b1c3d8ac989: No bootstrap required, opened a new log
I20260620 18:36:31.199467 30790 ts_tablet_manager.cc:1397] T 3bdabbd822b34f04a0f188f544221a94 P 856016baca2c4cf3b35c8b1c3d8ac989: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260620 18:36:31.200119 30791 ts_tablet_manager.cc:1428] T 3bdabbd822b34f04a0f188f544221a94 P 622c643eaee54366b09419fb11891fe9: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260620 18:36:31.201627 30790 raft_consensus.cc:348] T 3bdabbd822b34f04a0f188f544221a94 P 856016baca2c4cf3b35c8b1c3d8ac989 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b94701dfaaea4c6c95f229005e7b613e" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44231 } } peers { permanent_uuid: "856016baca2c4cf3b35c8b1c3d8ac989" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 46153 } } peers { permanent_uuid: "622c643eaee54366b09419fb11891fe9" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 38211 } }
I20260620 18:36:31.201943 30790 raft_consensus.cc:374] T 3bdabbd822b34f04a0f188f544221a94 P 856016baca2c4cf3b35c8b1c3d8ac989 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:36:31.202093 30790 raft_consensus.cc:729] T 3bdabbd822b34f04a0f188f544221a94 P 856016baca2c4cf3b35c8b1c3d8ac989 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 856016baca2c4cf3b35c8b1c3d8ac989, State: Initialized, Role: FOLLOWER
I20260620 18:36:31.202457 30790 consensus_queue.cc:260] T 3bdabbd822b34f04a0f188f544221a94 P 856016baca2c4cf3b35c8b1c3d8ac989 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b94701dfaaea4c6c95f229005e7b613e" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44231 } } peers { permanent_uuid: "856016baca2c4cf3b35c8b1c3d8ac989" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 46153 } } peers { permanent_uuid: "622c643eaee54366b09419fb11891fe9" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 38211 } }
I20260620 18:36:31.204902 30790 ts_tablet_manager.cc:1428] T 3bdabbd822b34f04a0f188f544221a94 P 856016baca2c4cf3b35c8b1c3d8ac989: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20260620 18:36:31.205691 30794 tablet_bootstrap.cc:492] T 3bdabbd822b34f04a0f188f544221a94 P b94701dfaaea4c6c95f229005e7b613e: Bootstrap starting.
I20260620 18:36:31.208060 30794 tablet_bootstrap.cc:654] T 3bdabbd822b34f04a0f188f544221a94 P b94701dfaaea4c6c95f229005e7b613e: Neither blocks nor log segments found. Creating new log.
I20260620 18:36:31.210204 30794 tablet_bootstrap.cc:492] T 3bdabbd822b34f04a0f188f544221a94 P b94701dfaaea4c6c95f229005e7b613e: No bootstrap required, opened a new log
I20260620 18:36:31.210371 30794 ts_tablet_manager.cc:1397] T 3bdabbd822b34f04a0f188f544221a94 P b94701dfaaea4c6c95f229005e7b613e: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 18:36:31.211380 30794 raft_consensus.cc:348] T 3bdabbd822b34f04a0f188f544221a94 P b94701dfaaea4c6c95f229005e7b613e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b94701dfaaea4c6c95f229005e7b613e" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44231 } } peers { permanent_uuid: "856016baca2c4cf3b35c8b1c3d8ac989" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 46153 } } peers { permanent_uuid: "622c643eaee54366b09419fb11891fe9" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 38211 } }
I20260620 18:36:31.211622 30794 raft_consensus.cc:374] T 3bdabbd822b34f04a0f188f544221a94 P b94701dfaaea4c6c95f229005e7b613e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:36:31.211761 30794 raft_consensus.cc:729] T 3bdabbd822b34f04a0f188f544221a94 P b94701dfaaea4c6c95f229005e7b613e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b94701dfaaea4c6c95f229005e7b613e, State: Initialized, Role: FOLLOWER
I20260620 18:36:31.212013 30794 consensus_queue.cc:260] T 3bdabbd822b34f04a0f188f544221a94 P b94701dfaaea4c6c95f229005e7b613e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b94701dfaaea4c6c95f229005e7b613e" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44231 } } peers { permanent_uuid: "856016baca2c4cf3b35c8b1c3d8ac989" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 46153 } } peers { permanent_uuid: "622c643eaee54366b09419fb11891fe9" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 38211 } }
I20260620 18:36:31.212522 30794 ts_tablet_manager.cc:1428] T 3bdabbd822b34f04a0f188f544221a94 P b94701dfaaea4c6c95f229005e7b613e: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260620 18:36:31.216802 30777 raft_consensus.cc:482] T 3bdabbd822b34f04a0f188f544221a94 P 856016baca2c4cf3b35c8b1c3d8ac989 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:36:31.217018 30777 raft_consensus.cc:504] T 3bdabbd822b34f04a0f188f544221a94 P 856016baca2c4cf3b35c8b1c3d8ac989 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b94701dfaaea4c6c95f229005e7b613e" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44231 } } peers { permanent_uuid: "856016baca2c4cf3b35c8b1c3d8ac989" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 46153 } } peers { permanent_uuid: "622c643eaee54366b09419fb11891fe9" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 38211 } }
I20260620 18:36:31.218698 30777 leader_election.cc:288] T 3bdabbd822b34f04a0f188f544221a94 P 856016baca2c4cf3b35c8b1c3d8ac989 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b94701dfaaea4c6c95f229005e7b613e (127.22.37.1:44231), 622c643eaee54366b09419fb11891fe9 (127.22.37.3:38211)
I20260620 18:36:31.227878 30752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:31.220467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.2:41569 (local address 127.22.37.3:38211)
0620 18:36:31.220622 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:31.220628 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:31.220776 (+ 148us) server_negotiation.cc:407] Connection header received
0620 18:36:31.221030 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:31.221036 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:31.221123 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:31.221217 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:36:31.221923 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:31.222843 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:31.225806 (+ 2963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:31.226711 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:31.226862 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:31.226993 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:31.227393 (+ 400us) server_negotiation.cc:299] Negotiation successful
0620 18:36:31.227536 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 18:36:31.228109 30774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:31.219265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.2:43745 (local address 127.22.37.1:44231)
0620 18:36:31.219454 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:31.219462 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:31.220132 (+ 670us) server_negotiation.cc:407] Connection header received
0620 18:36:31.220559 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:31.220564 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:31.220638 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:31.220708 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:36:31.221616 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:31.222818 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:31.225765 (+ 2947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:31.227073 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:31.227208 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:31.227343 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:31.227729 (+ 386us) server_negotiation.cc:299] Negotiation successful
0620 18:36:31.227855 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 18:36:31.228223 30796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:31.219062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.1:44231 (local address 127.22.37.2:43745)
0620 18:36:31.220017 (+ 955us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:31.220055 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:31.220358 (+ 303us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:31.220938 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:31.220945 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:31.220970 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:36:31.221426 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:31.221445 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:31.222963 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:31.222969 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:31.225603 (+ 2634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:31.225616 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:31.227319 (+ 1703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:31.227328 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:31.227547 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:31.227581 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 18:36:31.227679 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 18:36:31.227802 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":816,"mutex_wait_us":35,"thread_start_us":647,"threads_started":1}
I20260620 18:36:31.229099 30797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:31.220108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.3:38211 (local address 127.22.37.2:41569)
0620 18:36:31.220685 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:31.220703 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:31.220870 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:31.221258 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:31.221264 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:31.221296 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:36:31.221759 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:31.221767 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:31.223017 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:31.223037 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:31.225601 (+ 2564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:31.225616 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:31.226944 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:31.226951 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:31.227128 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:31.227190 (+ 62us) client_negotiation.cc:769] Sending connection context
0620 18:36:31.228617 (+ 1427us) client_negotiation.cc:240] Negotiation successful
0620 18:36:31.228753 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"mutex_wait_us":162,"thread_start_us":513,"threads_started":1}
I20260620 18:36:31.229198 30410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bdabbd822b34f04a0f188f544221a94" candidate_uuid: "856016baca2c4cf3b35c8b1c3d8ac989" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b94701dfaaea4c6c95f229005e7b613e" is_pre_election: true
I20260620 18:36:31.229687 30410 raft_consensus.cc:2454] T 3bdabbd822b34f04a0f188f544221a94 P b94701dfaaea4c6c95f229005e7b613e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 856016baca2c4cf3b35c8b1c3d8ac989 in term 0.
I20260620 18:36:31.230260 30685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bdabbd822b34f04a0f188f544221a94" candidate_uuid: "856016baca2c4cf3b35c8b1c3d8ac989" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "622c643eaee54366b09419fb11891fe9" is_pre_election: true
I20260620 18:36:31.230532 30685 raft_consensus.cc:2454] T 3bdabbd822b34f04a0f188f544221a94 P 622c643eaee54366b09419fb11891fe9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 856016baca2c4cf3b35c8b1c3d8ac989 in term 0.
I20260620 18:36:31.230607 30478 leader_election.cc:302] T 3bdabbd822b34f04a0f188f544221a94 P 856016baca2c4cf3b35c8b1c3d8ac989 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 856016baca2c4cf3b35c8b1c3d8ac989, b94701dfaaea4c6c95f229005e7b613e; no voters:
I20260620 18:36:31.231047 30777 raft_consensus.cc:2781] T 3bdabbd822b34f04a0f188f544221a94 P 856016baca2c4cf3b35c8b1c3d8ac989 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:36:31.231159 30777 raft_consensus.cc:482] T 3bdabbd822b34f04a0f188f544221a94 P 856016baca2c4cf3b35c8b1c3d8ac989 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:36:31.231272 30777 raft_consensus.cc:3037] T 3bdabbd822b34f04a0f188f544221a94 P 856016baca2c4cf3b35c8b1c3d8ac989 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:36:31.234287 30777 raft_consensus.cc:504] T 3bdabbd822b34f04a0f188f544221a94 P 856016baca2c4cf3b35c8b1c3d8ac989 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b94701dfaaea4c6c95f229005e7b613e" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44231 } } peers { permanent_uuid: "856016baca2c4cf3b35c8b1c3d8ac989" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 46153 } } peers { permanent_uuid: "622c643eaee54366b09419fb11891fe9" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 38211 } }
I20260620 18:36:31.234902 30777 leader_election.cc:288] T 3bdabbd822b34f04a0f188f544221a94 P 856016baca2c4cf3b35c8b1c3d8ac989 [CANDIDATE]: Term 1 election: Requested vote from peers b94701dfaaea4c6c95f229005e7b613e (127.22.37.1:44231), 622c643eaee54366b09419fb11891fe9 (127.22.37.3:38211)
I20260620 18:36:31.235364 30410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bdabbd822b34f04a0f188f544221a94" candidate_uuid: "856016baca2c4cf3b35c8b1c3d8ac989" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b94701dfaaea4c6c95f229005e7b613e"
I20260620 18:36:31.235421 30685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bdabbd822b34f04a0f188f544221a94" candidate_uuid: "856016baca2c4cf3b35c8b1c3d8ac989" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "622c643eaee54366b09419fb11891fe9"
I20260620 18:36:31.235597 30685 raft_consensus.cc:3037] T 3bdabbd822b34f04a0f188f544221a94 P 622c643eaee54366b09419fb11891fe9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:36:31.235612 30410 raft_consensus.cc:3037] T 3bdabbd822b34f04a0f188f544221a94 P b94701dfaaea4c6c95f229005e7b613e [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:36:31.238590 30685 raft_consensus.cc:2454] T 3bdabbd822b34f04a0f188f544221a94 P 622c643eaee54366b09419fb11891fe9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 856016baca2c4cf3b35c8b1c3d8ac989 in term 1.
I20260620 18:36:31.238889 30410 raft_consensus.cc:2454] T 3bdabbd822b34f04a0f188f544221a94 P b94701dfaaea4c6c95f229005e7b613e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 856016baca2c4cf3b35c8b1c3d8ac989 in term 1.
I20260620 18:36:31.239006 30478 leader_election.cc:302] T 3bdabbd822b34f04a0f188f544221a94 P 856016baca2c4cf3b35c8b1c3d8ac989 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 622c643eaee54366b09419fb11891fe9, 856016baca2c4cf3b35c8b1c3d8ac989; no voters:
I20260620 18:36:31.239301 30777 raft_consensus.cc:2781] T 3bdabbd822b34f04a0f188f544221a94 P 856016baca2c4cf3b35c8b1c3d8ac989 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:36:31.239845 30777 raft_consensus.cc:686] T 3bdabbd822b34f04a0f188f544221a94 P 856016baca2c4cf3b35c8b1c3d8ac989 [term 1 LEADER]: Becoming Leader. State: Replica: 856016baca2c4cf3b35c8b1c3d8ac989, State: Running, Role: LEADER
I20260620 18:36:31.240236 30777 consensus_queue.cc:237] T 3bdabbd822b34f04a0f188f544221a94 P 856016baca2c4cf3b35c8b1c3d8ac989 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b94701dfaaea4c6c95f229005e7b613e" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44231 } } peers { permanent_uuid: "856016baca2c4cf3b35c8b1c3d8ac989" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 46153 } } peers { permanent_uuid: "622c643eaee54366b09419fb11891fe9" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 38211 } }
I20260620 18:36:31.243729 30247 catalog_manager.cc:5495] T 3bdabbd822b34f04a0f188f544221a94 P 856016baca2c4cf3b35c8b1c3d8ac989 reported cstate change: term changed from 0 to 1, leader changed from <none> to 856016baca2c4cf3b35c8b1c3d8ac989 (127.22.37.2). New cstate: current_term: 1 leader_uuid: "856016baca2c4cf3b35c8b1c3d8ac989" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b94701dfaaea4c6c95f229005e7b613e" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "856016baca2c4cf3b35c8b1c3d8ac989" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 46153 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "622c643eaee54366b09419fb11891fe9" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 38211 } health_report { overall_health: UNKNOWN } } }
Jun 20 18:36:31 dist-test-slave-5lgp krb5kdc[30203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980588, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.37.62@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM
I20260620 18:36:31.299643 30805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:31.283972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:42963 (local address 127.0.0.1:43588)
0620 18:36:31.284774 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:31.284799 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:31.284943 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:31.285305 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:31.285308 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:31.286058 (+ 750us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:31.286288 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:31.286297 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:31.287831 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:31.287835 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:31.288847 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:31.288858 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:31.289845 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:31.289850 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:31.290008 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:31.290768 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:31.290789 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:31.292931 (+ 2142us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:36:31.296770 (+ 3839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:31.296780 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:31.296786 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:31.297224 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:31.297674 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:31.297679 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:31.297682 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:31.297877 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:31.298441 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:36:31.298450 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:36:31.298687 (+ 237us) client_negotiation.cc:769] Sending connection context
0620 18:36:31.298850 (+ 163us) client_negotiation.cc:240] Negotiation successful
0620 18:36:31.299198 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"mutex_wait_us":30,"thread_start_us":386,"threads_started":1}
I20260620 18:36:31.299714 30806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:31.283990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43588 (local address 127.22.37.62:42963)
0620 18:36:31.284987 (+ 997us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:31.284993 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:31.285018 (+ 25us) server_negotiation.cc:407] Connection header received
0620 18:36:31.285075 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:31.285081 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:31.285155 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:31.285277 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:31.286465 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:31.287690 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:31.288981 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:31.289654 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:31.289784 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:31.293143 (+ 3359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:31.293150 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:31.293155 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:31.293192 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:31.296635 (+ 3443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:31.297332 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:31.297338 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:31.297343 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:31.297550 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:31.298020 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:31.298025 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:31.298028 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:31.298291 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:31.298451 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:31.298922 (+ 471us) server_negotiation.cc:299] Negotiation successful
0620 18:36:31.299318 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":162,"server-negotiator.queue_time_us":875,"spinlock_wait_cycles":1408,"thread_start_us":702,"threads_started":1}
I20260620 18:36:31.304924 30247 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.22.37.62@KRBTEST.COM'} at 127.0.0.1:43588:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 18:36:31.306015 30247 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:36:31.312987 30385 tablet_service.cc:1467] Processing CreateTablet for tablet 97ba13f028454c658683f6424eb170ff (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=648c913fe5b3430b865f3538614ee1fc]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 18:36:31.312983 30524 tablet_service.cc:1467] Processing CreateTablet for tablet 97ba13f028454c658683f6424eb170ff (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=648c913fe5b3430b865f3538614ee1fc]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 18:36:31.313586 30385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97ba13f028454c658683f6424eb170ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:36:31.313587 30524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97ba13f028454c658683f6424eb170ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:36:31.317924 30662 tablet_service.cc:1467] Processing CreateTablet for tablet 97ba13f028454c658683f6424eb170ff (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=648c913fe5b3430b865f3538614ee1fc]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 18:36:31.318444 30662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97ba13f028454c658683f6424eb170ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:36:31.320750 30790 tablet_bootstrap.cc:492] T 97ba13f028454c658683f6424eb170ff P 856016baca2c4cf3b35c8b1c3d8ac989: Bootstrap starting.
I20260620 18:36:31.322216 30790 tablet_bootstrap.cc:654] T 97ba13f028454c658683f6424eb170ff P 856016baca2c4cf3b35c8b1c3d8ac989: Neither blocks nor log segments found. Creating new log.
I20260620 18:36:31.323027 30794 tablet_bootstrap.cc:492] T 97ba13f028454c658683f6424eb170ff P b94701dfaaea4c6c95f229005e7b613e: Bootstrap starting.
I20260620 18:36:31.324782 30794 tablet_bootstrap.cc:654] T 97ba13f028454c658683f6424eb170ff P b94701dfaaea4c6c95f229005e7b613e: Neither blocks nor log segments found. Creating new log.
I20260620 18:36:31.325901 30790 tablet_bootstrap.cc:492] T 97ba13f028454c658683f6424eb170ff P 856016baca2c4cf3b35c8b1c3d8ac989: No bootstrap required, opened a new log
I20260620 18:36:31.326068 30790 ts_tablet_manager.cc:1397] T 97ba13f028454c658683f6424eb170ff P 856016baca2c4cf3b35c8b1c3d8ac989: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260620 18:36:31.326949 30790 raft_consensus.cc:348] T 97ba13f028454c658683f6424eb170ff P 856016baca2c4cf3b35c8b1c3d8ac989 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b94701dfaaea4c6c95f229005e7b613e" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44231 } } peers { permanent_uuid: "856016baca2c4cf3b35c8b1c3d8ac989" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 46153 } } peers { permanent_uuid: "622c643eaee54366b09419fb11891fe9" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 38211 } }
I20260620 18:36:31.328329 30790 raft_consensus.cc:374] T 97ba13f028454c658683f6424eb170ff P 856016baca2c4cf3b35c8b1c3d8ac989 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:36:31.328470 30790 raft_consensus.cc:729] T 97ba13f028454c658683f6424eb170ff P 856016baca2c4cf3b35c8b1c3d8ac989 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 856016baca2c4cf3b35c8b1c3d8ac989, State: Initialized, Role: FOLLOWER
I20260620 18:36:31.328768 30790 consensus_queue.cc:260] T 97ba13f028454c658683f6424eb170ff P 856016baca2c4cf3b35c8b1c3d8ac989 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b94701dfaaea4c6c95f229005e7b613e" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44231 } } peers { permanent_uuid: "856016baca2c4cf3b35c8b1c3d8ac989" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 46153 } } peers { permanent_uuid: "622c643eaee54366b09419fb11891fe9" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 38211 } }
I20260620 18:36:31.329573 30790 ts_tablet_manager.cc:1428] T 97ba13f028454c658683f6424eb170ff P 856016baca2c4cf3b35c8b1c3d8ac989: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260620 18:36:31.329434 30777 tablet_replica.cc:439] T 97ba13f028454c658683f6424eb170ff P 856016baca2c4cf3b35c8b1c3d8ac989: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b94701dfaaea4c6c95f229005e7b613e" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44231 } } peers { permanent_uuid: "856016baca2c4cf3b35c8b1c3d8ac989" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 46153 } } peers { permanent_uuid: "622c643eaee54366b09419fb11891fe9" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 38211 } } }
I20260620 18:36:31.329804 30777 tablet_replica.cc:442] T 97ba13f028454c658683f6424eb170ff P 856016baca2c4cf3b35c8b1c3d8ac989: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:36:31.330150 30791 tablet_bootstrap.cc:492] T 97ba13f028454c658683f6424eb170ff P 622c643eaee54366b09419fb11891fe9: Bootstrap starting.
I20260620 18:36:31.331956 30791 tablet_bootstrap.cc:654] T 97ba13f028454c658683f6424eb170ff P 622c643eaee54366b09419fb11891fe9: Neither blocks nor log segments found. Creating new log.
I20260620 18:36:31.335515 30791 tablet_bootstrap.cc:492] T 97ba13f028454c658683f6424eb170ff P 622c643eaee54366b09419fb11891fe9: No bootstrap required, opened a new log
I20260620 18:36:31.335702 30791 ts_tablet_manager.cc:1397] T 97ba13f028454c658683f6424eb170ff P 622c643eaee54366b09419fb11891fe9: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.002s
I20260620 18:36:31.336359 30791 raft_consensus.cc:348] T 97ba13f028454c658683f6424eb170ff P 622c643eaee54366b09419fb11891fe9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b94701dfaaea4c6c95f229005e7b613e" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44231 } } peers { permanent_uuid: "856016baca2c4cf3b35c8b1c3d8ac989" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 46153 } } peers { permanent_uuid: "622c643eaee54366b09419fb11891fe9" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 38211 } }
I20260620 18:36:31.336627 30791 raft_consensus.cc:374] T 97ba13f028454c658683f6424eb170ff P 622c643eaee54366b09419fb11891fe9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:36:31.336709 30791 raft_consensus.cc:729] T 97ba13f028454c658683f6424eb170ff P 622c643eaee54366b09419fb11891fe9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 622c643eaee54366b09419fb11891fe9, State: Initialized, Role: FOLLOWER
I20260620 18:36:31.336994 30791 consensus_queue.cc:260] T 97ba13f028454c658683f6424eb170ff P 622c643eaee54366b09419fb11891fe9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b94701dfaaea4c6c95f229005e7b613e" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44231 } } peers { permanent_uuid: "856016baca2c4cf3b35c8b1c3d8ac989" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 46153 } } peers { permanent_uuid: "622c643eaee54366b09419fb11891fe9" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 38211 } }
I20260620 18:36:31.337556 30791 ts_tablet_manager.cc:1428] T 97ba13f028454c658683f6424eb170ff P 622c643eaee54366b09419fb11891fe9: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260620 18:36:31.337747 30778 tablet_replica.cc:439] T 97ba13f028454c658683f6424eb170ff P 622c643eaee54366b09419fb11891fe9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b94701dfaaea4c6c95f229005e7b613e" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44231 } } peers { permanent_uuid: "856016baca2c4cf3b35c8b1c3d8ac989" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 46153 } } peers { permanent_uuid: "622c643eaee54366b09419fb11891fe9" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 38211 } } }
I20260620 18:36:31.338099 30794 tablet_bootstrap.cc:492] T 97ba13f028454c658683f6424eb170ff P b94701dfaaea4c6c95f229005e7b613e: No bootstrap required, opened a new log
I20260620 18:36:31.338137 30778 tablet_replica.cc:442] T 97ba13f028454c658683f6424eb170ff P 622c643eaee54366b09419fb11891fe9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:36:31.338289 30794 ts_tablet_manager.cc:1397] T 97ba13f028454c658683f6424eb170ff P b94701dfaaea4c6c95f229005e7b613e: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.000s
I20260620 18:36:31.339067 30794 raft_consensus.cc:348] T 97ba13f028454c658683f6424eb170ff P b94701dfaaea4c6c95f229005e7b613e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b94701dfaaea4c6c95f229005e7b613e" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44231 } } peers { permanent_uuid: "856016baca2c4cf3b35c8b1c3d8ac989" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 46153 } } peers { permanent_uuid: "622c643eaee54366b09419fb11891fe9" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 38211 } }
I20260620 18:36:31.339362 30794 raft_consensus.cc:374] T 97ba13f028454c658683f6424eb170ff P b94701dfaaea4c6c95f229005e7b613e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:36:31.339437 30794 raft_consensus.cc:729] T 97ba13f028454c658683f6424eb170ff P b94701dfaaea4c6c95f229005e7b613e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b94701dfaaea4c6c95f229005e7b613e, State: Initialized, Role: FOLLOWER
I20260620 18:36:31.339675 30794 consensus_queue.cc:260] T 97ba13f028454c658683f6424eb170ff P b94701dfaaea4c6c95f229005e7b613e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b94701dfaaea4c6c95f229005e7b613e" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44231 } } peers { permanent_uuid: "856016baca2c4cf3b35c8b1c3d8ac989" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 46153 } } peers { permanent_uuid: "622c643eaee54366b09419fb11891fe9" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 38211 } }
I20260620 18:36:31.340219 30794 ts_tablet_manager.cc:1428] T 97ba13f028454c658683f6424eb170ff P b94701dfaaea4c6c95f229005e7b613e: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260620 18:36:31.343408 30768 tablet_replica.cc:439] T 97ba13f028454c658683f6424eb170ff P b94701dfaaea4c6c95f229005e7b613e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b94701dfaaea4c6c95f229005e7b613e" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44231 } } peers { permanent_uuid: "856016baca2c4cf3b35c8b1c3d8ac989" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 46153 } } peers { permanent_uuid: "622c643eaee54366b09419fb11891fe9" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 38211 } } }
I20260620 18:36:31.343775 30768 tablet_replica.cc:442] T 97ba13f028454c658683f6424eb170ff P b94701dfaaea4c6c95f229005e7b613e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:36:31.368539 30778 raft_consensus.cc:482] T 97ba13f028454c658683f6424eb170ff P 622c643eaee54366b09419fb11891fe9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:36:31.368806 30778 raft_consensus.cc:504] T 97ba13f028454c658683f6424eb170ff P 622c643eaee54366b09419fb11891fe9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b94701dfaaea4c6c95f229005e7b613e" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44231 } } peers { permanent_uuid: "856016baca2c4cf3b35c8b1c3d8ac989" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 46153 } } peers { permanent_uuid: "622c643eaee54366b09419fb11891fe9" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 38211 } }
I20260620 18:36:31.370334 30778 leader_election.cc:288] T 97ba13f028454c658683f6424eb170ff P 622c643eaee54366b09419fb11891fe9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b94701dfaaea4c6c95f229005e7b613e (127.22.37.1:44231), 856016baca2c4cf3b35c8b1c3d8ac989 (127.22.37.2:46153)
I20260620 18:36:31.378773 30774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:31.370650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.3:53305 (local address 127.22.37.1:44231)
0620 18:36:31.370876 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:31.370885 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:31.371436 (+ 551us) server_negotiation.cc:407] Connection header received
0620 18:36:31.371743 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:31.371749 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:31.371830 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:31.371895 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:36:31.372863 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:31.373957 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:31.376417 (+ 2460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:31.377726 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:31.377862 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:31.377962 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:31.378324 (+ 362us) server_negotiation.cc:299] Negotiation successful
0620 18:36:31.378476 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 18:36:31.378804 30810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:31.370656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.1:44231 (local address 127.22.37.3:53305)
0620 18:36:31.371345 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:31.371365 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:31.371549 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:31.372176 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:31.372184 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:31.372221 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:36:31.372700 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:31.372715 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:31.374090 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:31.374094 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:31.376259 (+ 2165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:31.376273 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:31.377923 (+ 1650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:31.377929 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:31.378097 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:31.378182 (+ 85us) client_negotiation.cc:769] Sending connection context
0620 18:36:31.378295 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 18:36:31.378440 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"mutex_wait_us":61,"thread_start_us":419,"threads_started":1}
I20260620 18:36:31.378896 30754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:31.371844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.3:52489 (local address 127.22.37.2:46153)
0620 18:36:31.372061 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:31.372067 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:31.372957 (+ 890us) server_negotiation.cc:407] Connection header received
0620 18:36:31.373121 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:31.373126 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:31.373236 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:31.373331 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:36:31.373892 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:31.374683 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:31.376467 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:31.377736 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:31.377862 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:31.377961 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:31.378479 (+ 518us) server_negotiation.cc:299] Negotiation successful
0620 18:36:31.378623 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260620 18:36:31.379702 30811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:31.371469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.2:46153 (local address 127.22.37.3:52489)
0620 18:36:31.372850 (+ 1381us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:31.372867 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:31.372992 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:31.373382 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:31.373386 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:31.373410 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:36:31.373740 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:31.373750 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:31.374812 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:31.374818 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:31.376254 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:31.376308 (+ 54us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:31.378107 (+ 1799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:31.378112 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:31.378309 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:31.378342 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 18:36:31.379267 (+ 925us) client_negotiation.cc:240] Negotiation successful
0620 18:36:31.379394 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1254,"mutex_wait_us":81,"thread_start_us":1041,"threads_started":1}
I20260620 18:36:31.379642 30410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97ba13f028454c658683f6424eb170ff" candidate_uuid: "622c643eaee54366b09419fb11891fe9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b94701dfaaea4c6c95f229005e7b613e" is_pre_election: true
I20260620 18:36:31.379910 30410 raft_consensus.cc:2454] T 97ba13f028454c658683f6424eb170ff P b94701dfaaea4c6c95f229005e7b613e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 622c643eaee54366b09419fb11891fe9 in term 0.
I20260620 18:36:31.380263 30549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97ba13f028454c658683f6424eb170ff" candidate_uuid: "622c643eaee54366b09419fb11891fe9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "856016baca2c4cf3b35c8b1c3d8ac989" is_pre_election: true
I20260620 18:36:31.380411 30617 leader_election.cc:302] T 97ba13f028454c658683f6424eb170ff P 622c643eaee54366b09419fb11891fe9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 622c643eaee54366b09419fb11891fe9, b94701dfaaea4c6c95f229005e7b613e; no voters:
I20260620 18:36:31.380501 30549 raft_consensus.cc:2454] T 97ba13f028454c658683f6424eb170ff P 856016baca2c4cf3b35c8b1c3d8ac989 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 622c643eaee54366b09419fb11891fe9 in term 0.
I20260620 18:36:31.380821 30778 raft_consensus.cc:2781] T 97ba13f028454c658683f6424eb170ff P 622c643eaee54366b09419fb11891fe9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:36:31.380960 30778 raft_consensus.cc:482] T 97ba13f028454c658683f6424eb170ff P 622c643eaee54366b09419fb11891fe9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:36:31.381083 30778 raft_consensus.cc:3037] T 97ba13f028454c658683f6424eb170ff P 622c643eaee54366b09419fb11891fe9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:36:31.383181 30778 raft_consensus.cc:504] T 97ba13f028454c658683f6424eb170ff P 622c643eaee54366b09419fb11891fe9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b94701dfaaea4c6c95f229005e7b613e" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44231 } } peers { permanent_uuid: "856016baca2c4cf3b35c8b1c3d8ac989" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 46153 } } peers { permanent_uuid: "622c643eaee54366b09419fb11891fe9" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 38211 } }
I20260620 18:36:31.383705 30778 leader_election.cc:288] T 97ba13f028454c658683f6424eb170ff P 622c643eaee54366b09419fb11891fe9 [CANDIDATE]: Term 1 election: Requested vote from peers b94701dfaaea4c6c95f229005e7b613e (127.22.37.1:44231), 856016baca2c4cf3b35c8b1c3d8ac989 (127.22.37.2:46153)
I20260620 18:36:31.384243 30549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97ba13f028454c658683f6424eb170ff" candidate_uuid: "622c643eaee54366b09419fb11891fe9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "856016baca2c4cf3b35c8b1c3d8ac989"
I20260620 18:36:31.384459 30549 raft_consensus.cc:3037] T 97ba13f028454c658683f6424eb170ff P 856016baca2c4cf3b35c8b1c3d8ac989 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:36:31.386620 30549 raft_consensus.cc:2454] T 97ba13f028454c658683f6424eb170ff P 856016baca2c4cf3b35c8b1c3d8ac989 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 622c643eaee54366b09419fb11891fe9 in term 1.
I20260620 18:36:31.386986 30617 leader_election.cc:302] T 97ba13f028454c658683f6424eb170ff P 622c643eaee54366b09419fb11891fe9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 622c643eaee54366b09419fb11891fe9, 856016baca2c4cf3b35c8b1c3d8ac989; no voters:
I20260620 18:36:31.386992 30410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97ba13f028454c658683f6424eb170ff" candidate_uuid: "622c643eaee54366b09419fb11891fe9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b94701dfaaea4c6c95f229005e7b613e"
I20260620 18:36:31.387223 30410 raft_consensus.cc:3037] T 97ba13f028454c658683f6424eb170ff P b94701dfaaea4c6c95f229005e7b613e [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:36:31.387274 30778 raft_consensus.cc:2781] T 97ba13f028454c658683f6424eb170ff P 622c643eaee54366b09419fb11891fe9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:36:31.387729 30778 raft_consensus.cc:686] T 97ba13f028454c658683f6424eb170ff P 622c643eaee54366b09419fb11891fe9 [term 1 LEADER]: Becoming Leader. State: Replica: 622c643eaee54366b09419fb11891fe9, State: Running, Role: LEADER
I20260620 18:36:31.388136 30778 consensus_queue.cc:237] T 97ba13f028454c658683f6424eb170ff P 622c643eaee54366b09419fb11891fe9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b94701dfaaea4c6c95f229005e7b613e" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44231 } } peers { permanent_uuid: "856016baca2c4cf3b35c8b1c3d8ac989" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 46153 } } peers { permanent_uuid: "622c643eaee54366b09419fb11891fe9" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 38211 } }
I20260620 18:36:31.390635 30410 raft_consensus.cc:2454] T 97ba13f028454c658683f6424eb170ff P b94701dfaaea4c6c95f229005e7b613e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 622c643eaee54366b09419fb11891fe9 in term 1.
I20260620 18:36:31.393321 30812 tablet_replica.cc:439] T 97ba13f028454c658683f6424eb170ff P 622c643eaee54366b09419fb11891fe9: TxnStatusTablet state changed. Reason: New leader 622c643eaee54366b09419fb11891fe9. Latest consensus state: current_term: 1 leader_uuid: "622c643eaee54366b09419fb11891fe9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b94701dfaaea4c6c95f229005e7b613e" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44231 } } peers { permanent_uuid: "856016baca2c4cf3b35c8b1c3d8ac989" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 46153 } } peers { permanent_uuid: "622c643eaee54366b09419fb11891fe9" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 38211 } } }
I20260620 18:36:31.393716 30812 tablet_replica.cc:442] T 97ba13f028454c658683f6424eb170ff P 622c643eaee54366b09419fb11891fe9: This TxnStatusTablet replica's current role is: LEADER
I20260620 18:36:31.393921 30247 catalog_manager.cc:5495] T 97ba13f028454c658683f6424eb170ff P 622c643eaee54366b09419fb11891fe9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 622c643eaee54366b09419fb11891fe9 (127.22.37.3). New cstate: current_term: 1 leader_uuid: "622c643eaee54366b09419fb11891fe9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b94701dfaaea4c6c95f229005e7b613e" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "856016baca2c4cf3b35c8b1c3d8ac989" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 46153 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "622c643eaee54366b09419fb11891fe9" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 38211 } health_report { overall_health: HEALTHY } } }
I20260620 18:36:31.399786 30814 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 18:36:31.400228 30814 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 18:36:31.425951 30805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:31.419385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.3:38211 (local address 127.0.0.1:44080)
0620 18:36:31.419591 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:31.419629 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:31.419793 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:31.420424 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:31.420429 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:31.420441 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 18:36:31.420625 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:31.420636 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:31.422189 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:31.422194 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:31.423649 (+ 1455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:31.423660 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:31.424427 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:31.424432 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:31.424588 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:31.424643 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 18:36:31.425393 (+ 750us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 18:36:31.425415 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 18:36:31.425508 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 18:36:31.425654 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260620 18:36:31.425989 30752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:31.419845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44080 (local address 127.22.37.3:38211)
0620 18:36:31.420083 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:31.420087 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:31.420102 (+ 15us) server_negotiation.cc:407] Connection header received
0620 18:36:31.420176 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:31.420179 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:31.420249 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:31.420405 (+ 156us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:36:31.420781 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:31.421992 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:31.423794 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:31.424278 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:31.424412 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:31.424828 (+ 416us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:36:31.425261 (+ 433us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:36:31.425440 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:31.425528 (+ 88us) server_negotiation.cc:299] Negotiation successful
0620 18:36:31.425622 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148}
I20260620 18:36:31.430307 30549 raft_consensus.cc:1261] T 97ba13f028454c658683f6424eb170ff P 856016baca2c4cf3b35c8b1c3d8ac989 [term 1 FOLLOWER]: Refusing update from remote peer 622c643eaee54366b09419fb11891fe9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:36:31.430588 30410 raft_consensus.cc:1261] T 97ba13f028454c658683f6424eb170ff P b94701dfaaea4c6c95f229005e7b613e [term 1 FOLLOWER]: Refusing update from remote peer 622c643eaee54366b09419fb11891fe9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:36:31.431260 30778 consensus_queue.cc:1035] T 97ba13f028454c658683f6424eb170ff P 622c643eaee54366b09419fb11891fe9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b94701dfaaea4c6c95f229005e7b613e" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:36:31.431888 30812 consensus_queue.cc:1035] T 97ba13f028454c658683f6424eb170ff P 622c643eaee54366b09419fb11891fe9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "856016baca2c4cf3b35c8b1c3d8ac989" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 46153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:36:31.443868 30768 tablet_replica.cc:439] T 97ba13f028454c658683f6424eb170ff P b94701dfaaea4c6c95f229005e7b613e: TxnStatusTablet state changed. Reason: New leader 622c643eaee54366b09419fb11891fe9. Latest consensus state: current_term: 1 leader_uuid: "622c643eaee54366b09419fb11891fe9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b94701dfaaea4c6c95f229005e7b613e" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44231 } } peers { permanent_uuid: "856016baca2c4cf3b35c8b1c3d8ac989" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 46153 } } peers { permanent_uuid: "622c643eaee54366b09419fb11891fe9" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 38211 } } }
I20260620 18:36:31.444240 30768 tablet_replica.cc:442] T 97ba13f028454c658683f6424eb170ff P b94701dfaaea4c6c95f229005e7b613e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:36:31.449875 30779 mvcc.cc:205] Tried to move back new op lower bound from 7298992502491033600 to 7298992502330335232. Current Snapshot: MvccSnapshot[applied={T|T < 7298992502491033600 or (T in {7298992502491033600})}]
I20260620 18:36:31.453433 30812 tablet_replica.cc:439] T 97ba13f028454c658683f6424eb170ff P 622c643eaee54366b09419fb11891fe9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "622c643eaee54366b09419fb11891fe9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b94701dfaaea4c6c95f229005e7b613e" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44231 } } peers { permanent_uuid: "856016baca2c4cf3b35c8b1c3d8ac989" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 46153 } } peers { permanent_uuid: "622c643eaee54366b09419fb11891fe9" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 38211 } } }
I20260620 18:36:31.454543 30812 tablet_replica.cc:442] T 97ba13f028454c658683f6424eb170ff P 622c643eaee54366b09419fb11891fe9: This TxnStatusTablet replica's current role is: LEADER
I20260620 18:36:31.457481 30768 tablet_replica.cc:439] T 97ba13f028454c658683f6424eb170ff P b94701dfaaea4c6c95f229005e7b613e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "622c643eaee54366b09419fb11891fe9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b94701dfaaea4c6c95f229005e7b613e" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44231 } } peers { permanent_uuid: "856016baca2c4cf3b35c8b1c3d8ac989" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 46153 } } peers { permanent_uuid: "622c643eaee54366b09419fb11891fe9" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 38211 } } }
I20260620 18:36:31.457830 30768 tablet_replica.cc:442] T 97ba13f028454c658683f6424eb170ff P b94701dfaaea4c6c95f229005e7b613e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:36:31.461794 30777 tablet_replica.cc:439] T 97ba13f028454c658683f6424eb170ff P 856016baca2c4cf3b35c8b1c3d8ac989: TxnStatusTablet state changed. Reason: New leader 622c643eaee54366b09419fb11891fe9. Latest consensus state: current_term: 1 leader_uuid: "622c643eaee54366b09419fb11891fe9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b94701dfaaea4c6c95f229005e7b613e" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44231 } } peers { permanent_uuid: "856016baca2c4cf3b35c8b1c3d8ac989" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 46153 } } peers { permanent_uuid: "622c643eaee54366b09419fb11891fe9" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 38211 } } }
I20260620 18:36:31.462117 30777 tablet_replica.cc:442] T 97ba13f028454c658683f6424eb170ff P 856016baca2c4cf3b35c8b1c3d8ac989: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:36:31.463693 30754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:31.455005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48600 (local address 127.22.37.2:46153)
0620 18:36:31.455167 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:31.455173 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:31.455192 (+ 19us) server_negotiation.cc:407] Connection header received
0620 18:36:31.455307 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:31.455313 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:31.455409 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:31.455564 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:36:31.456144 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:31.457240 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:31.460272 (+ 3032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:31.460905 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:31.461055 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:31.461780 (+ 725us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:36:31.462510 (+ 730us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:36:31.462664 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:31.463329 (+ 665us) server_negotiation.cc:299] Negotiation successful
0620 18:36:31.463414 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 18:36:31.465543 30777 tablet_replica.cc:439] T 97ba13f028454c658683f6424eb170ff P 856016baca2c4cf3b35c8b1c3d8ac989: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "622c643eaee54366b09419fb11891fe9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b94701dfaaea4c6c95f229005e7b613e" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44231 } } peers { permanent_uuid: "856016baca2c4cf3b35c8b1c3d8ac989" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 46153 } } peers { permanent_uuid: "622c643eaee54366b09419fb11891fe9" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 38211 } } }
I20260620 18:36:31.465816 30777 tablet_replica.cc:442] T 97ba13f028454c658683f6424eb170ff P 856016baca2c4cf3b35c8b1c3d8ac989: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:36:31.471390 30778 tablet_replica.cc:439] T 97ba13f028454c658683f6424eb170ff P 622c643eaee54366b09419fb11891fe9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "622c643eaee54366b09419fb11891fe9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b94701dfaaea4c6c95f229005e7b613e" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44231 } } peers { permanent_uuid: "856016baca2c4cf3b35c8b1c3d8ac989" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 46153 } } peers { permanent_uuid: "622c643eaee54366b09419fb11891fe9" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 38211 } } }
I20260620 18:36:31.471755 30778 tablet_replica.cc:442] T 97ba13f028454c658683f6424eb170ff P 622c643eaee54366b09419fb11891fe9: This TxnStatusTablet replica's current role is: LEADER
I20260620 18:36:31.507421 30752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:31.495845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.2:46143 (local address 127.22.37.3:38211)
0620 18:36:31.496020 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:31.496026 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:31.502044 (+ 6018us) server_negotiation.cc:407] Connection header received
0620 18:36:31.502253 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:31.502259 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:31.502357 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:31.502450 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:36:31.503204 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:31.503955 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:31.505532 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:31.506029 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:31.506211 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:31.506691 (+ 480us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:36:31.506802 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:36:31.506901 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:31.507012 (+ 111us) server_negotiation.cc:299] Negotiation successful
0620 18:36:31.507091 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 18:36:31.507793 30822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:31.495292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.3:38211 (local address 127.22.37.2:46143)
0620 18:36:31.501898 (+ 6606us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:31.501927 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:31.502106 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:31.502605 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:31.502610 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:31.502622 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 18:36:31.502982 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:31.502999 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:31.504103 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:31.504108 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:31.505336 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:31.505347 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:31.506248 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:31.506254 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:31.506432 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:31.506508 (+ 76us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 18:36:31.506903 (+ 395us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 18:36:31.506918 (+ 15us) client_negotiation.cc:769] Sending connection context
0620 18:36:31.506993 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 18:36:31.507124 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6392,"thread_start_us":6222,"threads_started":1}
I20260620 18:36:31.519037 30685 raft_consensus.cc:1261] T 3bdabbd822b34f04a0f188f544221a94 P 622c643eaee54366b09419fb11891fe9 [term 1 FOLLOWER]: Refusing update from remote peer 856016baca2c4cf3b35c8b1c3d8ac989: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:36:31.519037 30410 raft_consensus.cc:1261] T 3bdabbd822b34f04a0f188f544221a94 P b94701dfaaea4c6c95f229005e7b613e [term 1 FOLLOWER]: Refusing update from remote peer 856016baca2c4cf3b35c8b1c3d8ac989: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:36:31.520087 30798 consensus_queue.cc:1035] T 3bdabbd822b34f04a0f188f544221a94 P 856016baca2c4cf3b35c8b1c3d8ac989 [LEADER]: Connected to new peer: Peer: permanent_uuid: "622c643eaee54366b09419fb11891fe9" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 38211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:36:31.520953 30798 consensus_queue.cc:1035] T 3bdabbd822b34f04a0f188f544221a94 P 856016baca2c4cf3b35c8b1c3d8ac989 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b94701dfaaea4c6c95f229005e7b613e" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:36:31.583705 30754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:31.575124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.3:58509 (local address 127.22.37.2:46153)
0620 18:36:31.575276 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:31.575280 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:31.577075 (+ 1795us) server_negotiation.cc:407] Connection header received
0620 18:36:31.577562 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:31.577569 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:31.577658 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:31.577735 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:36:31.578394 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:31.579108 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:31.581903 (+ 2795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:31.582382 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:31.582525 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:31.582960 (+ 435us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:36:31.583067 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:36:31.583163 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:31.583306 (+ 143us) server_negotiation.cc:299] Negotiation successful
0620 18:36:31.583380 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 18:36:31.583858 30827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:31.575218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.2:46153 (local address 127.22.37.3:58509)
0620 18:36:31.576623 (+ 1405us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:31.576642 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:31.577201 (+ 559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:31.577930 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:31.577940 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:31.577950 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 18:36:31.578246 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:31.578254 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:31.579252 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:31.579259 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:31.581768 (+ 2509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:31.581781 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:31.582569 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:31.582578 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:31.582716 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:31.582773 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 18:36:31.583164 (+ 391us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 18:36:31.583183 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 18:36:31.583286 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 18:36:31.583429 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":759,"thread_start_us":599,"threads_started":1}
I20260620 18:36:31.642313 30248 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43586:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:36:31.642575 30248 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43586:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:36:31.646505 30248 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4d9466ac6e3e4d1c95493d9373c03f45: Sending DeleteTablet for 3 replicas of tablet 3bdabbd822b34f04a0f188f544221a94
I20260620 18:36:31.647666 30662 tablet_service.cc:1514] Processing DeleteTablet for tablet 3bdabbd822b34f04a0f188f544221a94 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:36:31 UTC) from {username='oryx', principal='oryx/127.22.37.62@KRBTEST.COM'} at 127.0.0.1:44078
I20260620 18:36:31.647985 30787 tablet_replica.cc:330] T 3bdabbd822b34f04a0f188f544221a94 P 622c643eaee54366b09419fb11891fe9: stopping tablet replica
I20260620 18:36:31.648196 30524 tablet_service.cc:1514] Processing DeleteTablet for tablet 3bdabbd822b34f04a0f188f544221a94 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:36:31 UTC) from {username='oryx', principal='oryx/127.22.37.62@KRBTEST.COM'} at 127.0.0.1:48596
I20260620 18:36:31.648244 30787 raft_consensus.cc:2229] T 3bdabbd822b34f04a0f188f544221a94 P 622c643eaee54366b09419fb11891fe9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:36:31.648470 30787 raft_consensus.cc:2258] T 3bdabbd822b34f04a0f188f544221a94 P 622c643eaee54366b09419fb11891fe9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:36:31.649080 30789 tablet_replica.cc:330] T 3bdabbd822b34f04a0f188f544221a94 P 856016baca2c4cf3b35c8b1c3d8ac989: stopping tablet replica
I20260620 18:36:31.649281 30789 raft_consensus.cc:2229] T 3bdabbd822b34f04a0f188f544221a94 P 856016baca2c4cf3b35c8b1c3d8ac989 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:36:31.649847 30789 raft_consensus.cc:2258] T 3bdabbd822b34f04a0f188f544221a94 P 856016baca2c4cf3b35c8b1c3d8ac989 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:36:31.649947 30787 ts_tablet_manager.cc:1905] T 3bdabbd822b34f04a0f188f544221a94 P 622c643eaee54366b09419fb11891fe9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:36:31.650477 30385 tablet_service.cc:1514] Processing DeleteTablet for tablet 3bdabbd822b34f04a0f188f544221a94 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:36:31 UTC) from {username='oryx', principal='oryx/127.22.37.62@KRBTEST.COM'} at 127.0.0.1:55434
I20260620 18:36:31.650993 30789 ts_tablet_manager.cc:1905] T 3bdabbd822b34f04a0f188f544221a94 P 856016baca2c4cf3b35c8b1c3d8ac989: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:36:31.653599 30788 tablet_replica.cc:330] T 3bdabbd822b34f04a0f188f544221a94 P b94701dfaaea4c6c95f229005e7b613e: stopping tablet replica
I20260620 18:36:31.653782 30788 raft_consensus.cc:2229] T 3bdabbd822b34f04a0f188f544221a94 P b94701dfaaea4c6c95f229005e7b613e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:36:31.654000 30788 raft_consensus.cc:2258] T 3bdabbd822b34f04a0f188f544221a94 P b94701dfaaea4c6c95f229005e7b613e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:36:31.655316 30788 ts_tablet_manager.cc:1905] T 3bdabbd822b34f04a0f188f544221a94 P b94701dfaaea4c6c95f229005e7b613e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:36:31.656759 30789 ts_tablet_manager.cc:1918] T 3bdabbd822b34f04a0f188f544221a94 P 856016baca2c4cf3b35c8b1c3d8ac989: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:36:31.656880 30789 log.cc:1196] T 3bdabbd822b34f04a0f188f544221a94 P 856016baca2c4cf3b35c8b1c3d8ac989: Deleting WAL directory at /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/minicluster-data/ts-1/wal/wals/3bdabbd822b34f04a0f188f544221a94
I20260620 18:36:31.657101 30787 ts_tablet_manager.cc:1918] T 3bdabbd822b34f04a0f188f544221a94 P 622c643eaee54366b09419fb11891fe9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:36:31.657248 30787 log.cc:1196] T 3bdabbd822b34f04a0f188f544221a94 P 622c643eaee54366b09419fb11891fe9: Deleting WAL directory at /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/minicluster-data/ts-2/wal/wals/3bdabbd822b34f04a0f188f544221a94
I20260620 18:36:31.657267 30789 ts_tablet_manager.cc:1939] T 3bdabbd822b34f04a0f188f544221a94 P 856016baca2c4cf3b35c8b1c3d8ac989: Deleting consensus metadata
I20260620 18:36:31.657680 30787 ts_tablet_manager.cc:1939] T 3bdabbd822b34f04a0f188f544221a94 P 622c643eaee54366b09419fb11891fe9: Deleting consensus metadata
I20260620 18:36:31.659039 30235 catalog_manager.cc:4841] TS 856016baca2c4cf3b35c8b1c3d8ac989 (127.22.37.2:46153): tablet 3bdabbd822b34f04a0f188f544221a94 (table test-table [id=032b0a3a56f44eba9e72e979df60ec00]) successfully deleted
I20260620 18:36:31.662981 30788 ts_tablet_manager.cc:1918] T 3bdabbd822b34f04a0f188f544221a94 P b94701dfaaea4c6c95f229005e7b613e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:36:31.663254 30788 log.cc:1196] T 3bdabbd822b34f04a0f188f544221a94 P b94701dfaaea4c6c95f229005e7b613e: Deleting WAL directory at /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/minicluster-data/ts-0/wal/wals/3bdabbd822b34f04a0f188f544221a94
I20260620 18:36:31.663950 30788 ts_tablet_manager.cc:1939] T 3bdabbd822b34f04a0f188f544221a94 P b94701dfaaea4c6c95f229005e7b613e: Deleting consensus metadata
I20260620 18:36:31.670245 30235 catalog_manager.cc:4841] TS 622c643eaee54366b09419fb11891fe9 (127.22.37.3:38211): tablet 3bdabbd822b34f04a0f188f544221a94 (table test-table [id=032b0a3a56f44eba9e72e979df60ec00]) successfully deleted
I20260620 18:36:31.671372 30235 catalog_manager.cc:4841] TS b94701dfaaea4c6c95f229005e7b613e (127.22.37.1:44231): tablet 3bdabbd822b34f04a0f188f544221a94 (table test-table [id=032b0a3a56f44eba9e72e979df60ec00]) successfully deleted
Jun 20 18:36:31 dist-test-slave-5lgp krb5kdc[30203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980591, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 18:36:31 dist-test-slave-5lgp krb5kdc[30203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:36:31 dist-test-slave-5lgp krb5kdc[30203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 18:36:31.770902 30806 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:36:31.747102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43602 (local address 127.22.37.62:42963)
0620 18:36:31.747371 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:31.747378 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:31.759254 (+ 11876us) server_negotiation.cc:407] Connection header received
0620 18:36:31.759481 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:31.759487 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:31.759579 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:31.759683 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:31.761407 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:31.762431 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:31.764374 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:31.765016 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:31.765106 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:31.770644 (+ 5538us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43602: BlockingRecv error: recv got EOF from 127.0.0.1:43602 (error 108)
Metrics: {"server-negotiator.queue_time_us":125}
Jun 20 18:36:31 dist-test-slave-5lgp krb5kdc[30203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980591, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM
I20260620 18:36:31.802733 30806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:31.774383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43614 (local address 127.22.37.62:42963)
0620 18:36:31.774602 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:31.774611 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:31.775033 (+ 422us) server_negotiation.cc:407] Connection header received
0620 18:36:31.785545 (+ 10512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:31.785551 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:31.785644 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:31.785764 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:31.787486 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:31.788641 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:31.790571 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:31.791273 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:31.791360 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:31.796418 (+ 5058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:31.796425 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:31.796430 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:31.796470 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:31.799956 (+ 3486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:31.800689 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:31.800694 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:31.800698 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:31.800834 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:31.801227 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:31.801232 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:31.801235 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:31.801582 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:31.801841 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:31.802132 (+ 291us) server_negotiation.cc:299] Negotiation successful
0620 18:36:31.802378 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 18:36:31.806294 30248 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43614:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:36:31.806922 30248 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:36:31.814426 30524 tablet_service.cc:1467] Processing CreateTablet for tablet ce657fac4eb442afb5ede23bd081165a (DEFAULT_TABLE table=test-table [id=aa58a77479034122a8f5813a76ec0fa4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:36:31.814853 30524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce657fac4eb442afb5ede23bd081165a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:36:31.820451 30790 tablet_bootstrap.cc:492] T ce657fac4eb442afb5ede23bd081165a P 856016baca2c4cf3b35c8b1c3d8ac989: Bootstrap starting.
I20260620 18:36:31.820801 30662 tablet_service.cc:1467] Processing CreateTablet for tablet ce657fac4eb442afb5ede23bd081165a (DEFAULT_TABLE table=test-table [id=aa58a77479034122a8f5813a76ec0fa4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:36:31.821233 30662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce657fac4eb442afb5ede23bd081165a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:36:31.821466 30385 tablet_service.cc:1467] Processing CreateTablet for tablet ce657fac4eb442afb5ede23bd081165a (DEFAULT_TABLE table=test-table [id=aa58a77479034122a8f5813a76ec0fa4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:36:31.821870 30385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce657fac4eb442afb5ede23bd081165a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:36:31.822149 30790 tablet_bootstrap.cc:654] T ce657fac4eb442afb5ede23bd081165a P 856016baca2c4cf3b35c8b1c3d8ac989: Neither blocks nor log segments found. Creating new log.
I20260620 18:36:31.835569 30790 tablet_bootstrap.cc:492] T ce657fac4eb442afb5ede23bd081165a P 856016baca2c4cf3b35c8b1c3d8ac989: No bootstrap required, opened a new log
I20260620 18:36:31.835831 30790 ts_tablet_manager.cc:1397] T ce657fac4eb442afb5ede23bd081165a P 856016baca2c4cf3b35c8b1c3d8ac989: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.007s
I20260620 18:36:31.836752 30790 raft_consensus.cc:348] T ce657fac4eb442afb5ede23bd081165a P 856016baca2c4cf3b35c8b1c3d8ac989 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b94701dfaaea4c6c95f229005e7b613e" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44231 } } peers { permanent_uuid: "856016baca2c4cf3b35c8b1c3d8ac989" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 46153 } } peers { permanent_uuid: "622c643eaee54366b09419fb11891fe9" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 38211 } }
I20260620 18:36:31.837018 30794 tablet_bootstrap.cc:492] T ce657fac4eb442afb5ede23bd081165a P b94701dfaaea4c6c95f229005e7b613e: Bootstrap starting.
I20260620 18:36:31.837158 30790 raft_consensus.cc:374] T ce657fac4eb442afb5ede23bd081165a P 856016baca2c4cf3b35c8b1c3d8ac989 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:36:31.837319 30790 raft_consensus.cc:729] T ce657fac4eb442afb5ede23bd081165a P 856016baca2c4cf3b35c8b1c3d8ac989 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 856016baca2c4cf3b35c8b1c3d8ac989, State: Initialized, Role: FOLLOWER
I20260620 18:36:31.838707 30794 tablet_bootstrap.cc:654] T ce657fac4eb442afb5ede23bd081165a P b94701dfaaea4c6c95f229005e7b613e: Neither blocks nor log segments found. Creating new log.
I20260620 18:36:31.837659 30790 consensus_queue.cc:260] T ce657fac4eb442afb5ede23bd081165a P 856016baca2c4cf3b35c8b1c3d8ac989 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b94701dfaaea4c6c95f229005e7b613e" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44231 } } peers { permanent_uuid: "856016baca2c4cf3b35c8b1c3d8ac989" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 46153 } } peers { permanent_uuid: "622c643eaee54366b09419fb11891fe9" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 38211 } }
I20260620 18:36:31.842607 30790 ts_tablet_manager.cc:1428] T ce657fac4eb442afb5ede23bd081165a P 856016baca2c4cf3b35c8b1c3d8ac989: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20260620 18:36:31.843295 30791 tablet_bootstrap.cc:492] T ce657fac4eb442afb5ede23bd081165a P 622c643eaee54366b09419fb11891fe9: Bootstrap starting.
I20260620 18:36:31.845110 30791 tablet_bootstrap.cc:654] T ce657fac4eb442afb5ede23bd081165a P 622c643eaee54366b09419fb11891fe9: Neither blocks nor log segments found. Creating new log.
I20260620 18:36:31.847523 30791 tablet_bootstrap.cc:492] T ce657fac4eb442afb5ede23bd081165a P 622c643eaee54366b09419fb11891fe9: No bootstrap required, opened a new log
I20260620 18:36:31.847692 30791 ts_tablet_manager.cc:1397] T ce657fac4eb442afb5ede23bd081165a P 622c643eaee54366b09419fb11891fe9: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260620 18:36:31.848289 30791 raft_consensus.cc:348] T ce657fac4eb442afb5ede23bd081165a P 622c643eaee54366b09419fb11891fe9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b94701dfaaea4c6c95f229005e7b613e" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44231 } } peers { permanent_uuid: "856016baca2c4cf3b35c8b1c3d8ac989" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 46153 } } peers { permanent_uuid: "622c643eaee54366b09419fb11891fe9" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 38211 } }
I20260620 18:36:31.848521 30791 raft_consensus.cc:374] T ce657fac4eb442afb5ede23bd081165a P 622c643eaee54366b09419fb11891fe9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:36:31.848644 30791 raft_consensus.cc:729] T ce657fac4eb442afb5ede23bd081165a P 622c643eaee54366b09419fb11891fe9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 622c643eaee54366b09419fb11891fe9, State: Initialized, Role: FOLLOWER
I20260620 18:36:31.848875 30791 consensus_queue.cc:260] T ce657fac4eb442afb5ede23bd081165a P 622c643eaee54366b09419fb11891fe9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b94701dfaaea4c6c95f229005e7b613e" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44231 } } peers { permanent_uuid: "856016baca2c4cf3b35c8b1c3d8ac989" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 46153 } } peers { permanent_uuid: "622c643eaee54366b09419fb11891fe9" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 38211 } }
I20260620 18:36:31.849453 30791 ts_tablet_manager.cc:1428] T ce657fac4eb442afb5ede23bd081165a P 622c643eaee54366b09419fb11891fe9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 18:36:31.855450 30794 tablet_bootstrap.cc:492] T ce657fac4eb442afb5ede23bd081165a P b94701dfaaea4c6c95f229005e7b613e: No bootstrap required, opened a new log
I20260620 18:36:31.855628 30794 ts_tablet_manager.cc:1397] T ce657fac4eb442afb5ede23bd081165a P b94701dfaaea4c6c95f229005e7b613e: Time spent bootstrapping tablet: real 0.019s user 0.001s sys 0.010s
I20260620 18:36:31.856254 30794 raft_consensus.cc:348] T ce657fac4eb442afb5ede23bd081165a P b94701dfaaea4c6c95f229005e7b613e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b94701dfaaea4c6c95f229005e7b613e" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44231 } } peers { permanent_uuid: "856016baca2c4cf3b35c8b1c3d8ac989" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 46153 } } peers { permanent_uuid: "622c643eaee54366b09419fb11891fe9" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 38211 } }
I20260620 18:36:31.856483 30794 raft_consensus.cc:374] T ce657fac4eb442afb5ede23bd081165a P b94701dfaaea4c6c95f229005e7b613e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:36:31.856580 30794 raft_consensus.cc:729] T ce657fac4eb442afb5ede23bd081165a P b94701dfaaea4c6c95f229005e7b613e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b94701dfaaea4c6c95f229005e7b613e, State: Initialized, Role: FOLLOWER
I20260620 18:36:31.856827 30794 consensus_queue.cc:260] T ce657fac4eb442afb5ede23bd081165a P b94701dfaaea4c6c95f229005e7b613e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b94701dfaaea4c6c95f229005e7b613e" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44231 } } peers { permanent_uuid: "856016baca2c4cf3b35c8b1c3d8ac989" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 46153 } } peers { permanent_uuid: "622c643eaee54366b09419fb11891fe9" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 38211 } }
I20260620 18:36:31.857389 30794 ts_tablet_manager.cc:1428] T ce657fac4eb442afb5ede23bd081165a P b94701dfaaea4c6c95f229005e7b613e: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260620 18:36:31.995458 30768 raft_consensus.cc:482] T ce657fac4eb442afb5ede23bd081165a P b94701dfaaea4c6c95f229005e7b613e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:36:31.995667 30768 raft_consensus.cc:504] T ce657fac4eb442afb5ede23bd081165a P b94701dfaaea4c6c95f229005e7b613e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b94701dfaaea4c6c95f229005e7b613e" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44231 } } peers { permanent_uuid: "856016baca2c4cf3b35c8b1c3d8ac989" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 46153 } } peers { permanent_uuid: "622c643eaee54366b09419fb11891fe9" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 38211 } }
I20260620 18:36:31.996372 30768 leader_election.cc:288] T ce657fac4eb442afb5ede23bd081165a P b94701dfaaea4c6c95f229005e7b613e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 856016baca2c4cf3b35c8b1c3d8ac989 (127.22.37.2:46153), 622c643eaee54366b09419fb11891fe9 (127.22.37.3:38211)
I20260620 18:36:31.996995 30549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce657fac4eb442afb5ede23bd081165a" candidate_uuid: "b94701dfaaea4c6c95f229005e7b613e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "856016baca2c4cf3b35c8b1c3d8ac989" is_pre_election: true
I20260620 18:36:31.997118 30684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce657fac4eb442afb5ede23bd081165a" candidate_uuid: "b94701dfaaea4c6c95f229005e7b613e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "622c643eaee54366b09419fb11891fe9" is_pre_election: true
I20260620 18:36:31.997244 30549 raft_consensus.cc:2454] T ce657fac4eb442afb5ede23bd081165a P 856016baca2c4cf3b35c8b1c3d8ac989 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b94701dfaaea4c6c95f229005e7b613e in term 0.
I20260620 18:36:31.997431 30684 raft_consensus.cc:2454] T ce657fac4eb442afb5ede23bd081165a P 622c643eaee54366b09419fb11891fe9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b94701dfaaea4c6c95f229005e7b613e in term 0.
I20260620 18:36:31.997810 30338 leader_election.cc:302] T ce657fac4eb442afb5ede23bd081165a P b94701dfaaea4c6c95f229005e7b613e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 856016baca2c4cf3b35c8b1c3d8ac989, b94701dfaaea4c6c95f229005e7b613e; no voters:
I20260620 18:36:31.998121 30768 raft_consensus.cc:2781] T ce657fac4eb442afb5ede23bd081165a P b94701dfaaea4c6c95f229005e7b613e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:36:31.998219 30768 raft_consensus.cc:482] T ce657fac4eb442afb5ede23bd081165a P b94701dfaaea4c6c95f229005e7b613e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:36:31.998317 30768 raft_consensus.cc:3037] T ce657fac4eb442afb5ede23bd081165a P b94701dfaaea4c6c95f229005e7b613e [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:36:32.001194 30768 raft_consensus.cc:504] T ce657fac4eb442afb5ede23bd081165a P b94701dfaaea4c6c95f229005e7b613e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b94701dfaaea4c6c95f229005e7b613e" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44231 } } peers { permanent_uuid: "856016baca2c4cf3b35c8b1c3d8ac989" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 46153 } } peers { permanent_uuid: "622c643eaee54366b09419fb11891fe9" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 38211 } }
I20260620 18:36:32.001873 30768 leader_election.cc:288] T ce657fac4eb442afb5ede23bd081165a P b94701dfaaea4c6c95f229005e7b613e [CANDIDATE]: Term 1 election: Requested vote from peers 856016baca2c4cf3b35c8b1c3d8ac989 (127.22.37.2:46153), 622c643eaee54366b09419fb11891fe9 (127.22.37.3:38211)
I20260620 18:36:32.002418 30684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce657fac4eb442afb5ede23bd081165a" candidate_uuid: "b94701dfaaea4c6c95f229005e7b613e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "622c643eaee54366b09419fb11891fe9"
I20260620 18:36:32.002454 30549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce657fac4eb442afb5ede23bd081165a" candidate_uuid: "b94701dfaaea4c6c95f229005e7b613e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "856016baca2c4cf3b35c8b1c3d8ac989"
I20260620 18:36:32.002609 30684 raft_consensus.cc:3037] T ce657fac4eb442afb5ede23bd081165a P 622c643eaee54366b09419fb11891fe9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:36:32.002652 30549 raft_consensus.cc:3037] T ce657fac4eb442afb5ede23bd081165a P 856016baca2c4cf3b35c8b1c3d8ac989 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:36:32.005759 30549 raft_consensus.cc:2454] T ce657fac4eb442afb5ede23bd081165a P 856016baca2c4cf3b35c8b1c3d8ac989 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b94701dfaaea4c6c95f229005e7b613e in term 1.
I20260620 18:36:32.006246 30338 leader_election.cc:302] T ce657fac4eb442afb5ede23bd081165a P b94701dfaaea4c6c95f229005e7b613e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 856016baca2c4cf3b35c8b1c3d8ac989, b94701dfaaea4c6c95f229005e7b613e; no voters:
I20260620 18:36:32.006476 30684 raft_consensus.cc:2454] T ce657fac4eb442afb5ede23bd081165a P 622c643eaee54366b09419fb11891fe9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b94701dfaaea4c6c95f229005e7b613e in term 1.
I20260620 18:36:32.007241 30768 raft_consensus.cc:2781] T ce657fac4eb442afb5ede23bd081165a P b94701dfaaea4c6c95f229005e7b613e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:36:32.009624 30768 raft_consensus.cc:686] T ce657fac4eb442afb5ede23bd081165a P b94701dfaaea4c6c95f229005e7b613e [term 1 LEADER]: Becoming Leader. State: Replica: b94701dfaaea4c6c95f229005e7b613e, State: Running, Role: LEADER
I20260620 18:36:32.009943 30768 consensus_queue.cc:237] T ce657fac4eb442afb5ede23bd081165a P b94701dfaaea4c6c95f229005e7b613e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b94701dfaaea4c6c95f229005e7b613e" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44231 } } peers { permanent_uuid: "856016baca2c4cf3b35c8b1c3d8ac989" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 46153 } } peers { permanent_uuid: "622c643eaee54366b09419fb11891fe9" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 38211 } }
I20260620 18:36:32.015241 30248 catalog_manager.cc:5495] T ce657fac4eb442afb5ede23bd081165a P b94701dfaaea4c6c95f229005e7b613e reported cstate change: term changed from 0 to 1, leader changed from <none> to b94701dfaaea4c6c95f229005e7b613e (127.22.37.1). New cstate: current_term: 1 leader_uuid: "b94701dfaaea4c6c95f229005e7b613e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b94701dfaaea4c6c95f229005e7b613e" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "856016baca2c4cf3b35c8b1c3d8ac989" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 46153 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "622c643eaee54366b09419fb11891fe9" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 38211 } health_report { overall_health: UNKNOWN } } }
I20260620 18:36:32.064749 30848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:32.058867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55444 (local address 127.22.37.1:44231)
0620 18:36:32.059416 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:32.059424 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:32.059447 (+ 23us) server_negotiation.cc:407] Connection header received
0620 18:36:32.059520 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:32.059525 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:32.059613 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:32.059732 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:36:32.060066 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:32.061258 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:32.062739 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:32.063409 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:32.063545 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:32.063969 (+ 424us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:36:32.064096 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:36:32.064190 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:32.064338 (+ 148us) server_negotiation.cc:299] Negotiation successful
0620 18:36:32.064421 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":254,"threads_started":1}
I20260620 18:36:32.067536 30684 raft_consensus.cc:1261] T ce657fac4eb442afb5ede23bd081165a P 622c643eaee54366b09419fb11891fe9 [term 1 FOLLOWER]: Refusing update from remote peer b94701dfaaea4c6c95f229005e7b613e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:36:32.067649 30549 raft_consensus.cc:1261] T ce657fac4eb442afb5ede23bd081165a P 856016baca2c4cf3b35c8b1c3d8ac989 [term 1 FOLLOWER]: Refusing update from remote peer b94701dfaaea4c6c95f229005e7b613e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:36:32.068245 30768 consensus_queue.cc:1035] T ce657fac4eb442afb5ede23bd081165a P b94701dfaaea4c6c95f229005e7b613e [LEADER]: Connected to new peer: Peer: permanent_uuid: "856016baca2c4cf3b35c8b1c3d8ac989" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 46153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:36:32.068608 30845 consensus_queue.cc:1035] T ce657fac4eb442afb5ede23bd081165a P b94701dfaaea4c6c95f229005e7b613e [LEADER]: Connected to new peer: Peer: permanent_uuid: "622c643eaee54366b09419fb11891fe9" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 38211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:36:32.073302 30776 mvcc.cc:205] Tried to move back new op lower bound from 7298992505103040512 to 7298992504875130880. Current Snapshot: MvccSnapshot[applied={T|T < 7298992505103040512}]
I20260620 18:36:32.083449 30248 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43614:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:36:32.083721 30248 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43614:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:36:32.089727 30385 tablet_service.cc:1514] Processing DeleteTablet for tablet ce657fac4eb442afb5ede23bd081165a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:36:32 UTC) from {username='oryx', principal='oryx/127.22.37.62@KRBTEST.COM'} at 127.0.0.1:55434
I20260620 18:36:32.090070 30788 tablet_replica.cc:330] T ce657fac4eb442afb5ede23bd081165a P b94701dfaaea4c6c95f229005e7b613e: stopping tablet replica
I20260620 18:36:32.090277 30788 raft_consensus.cc:2229] T ce657fac4eb442afb5ede23bd081165a P b94701dfaaea4c6c95f229005e7b613e [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:36:32.090960 30788 raft_consensus.cc:2258] T ce657fac4eb442afb5ede23bd081165a P b94701dfaaea4c6c95f229005e7b613e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:36:32.091928 30788 ts_tablet_manager.cc:1905] T ce657fac4eb442afb5ede23bd081165a P b94701dfaaea4c6c95f229005e7b613e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:36:32.091981 30524 tablet_service.cc:1514] Processing DeleteTablet for tablet ce657fac4eb442afb5ede23bd081165a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:36:32 UTC) from {username='oryx', principal='oryx/127.22.37.62@KRBTEST.COM'} at 127.0.0.1:48596
I20260620 18:36:32.092316 30789 tablet_replica.cc:330] T ce657fac4eb442afb5ede23bd081165a P 856016baca2c4cf3b35c8b1c3d8ac989: stopping tablet replica
I20260620 18:36:32.092502 30789 raft_consensus.cc:2229] T ce657fac4eb442afb5ede23bd081165a P 856016baca2c4cf3b35c8b1c3d8ac989 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:36:32.092721 30789 raft_consensus.cc:2258] T ce657fac4eb442afb5ede23bd081165a P 856016baca2c4cf3b35c8b1c3d8ac989 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:36:32.093194 30248 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4d9466ac6e3e4d1c95493d9373c03f45: Sending DeleteTablet for 3 replicas of tablet ce657fac4eb442afb5ede23bd081165a
I20260620 18:36:32.093892 30385 tablet_service.cc:1514] Processing DeleteTablet for tablet ce657fac4eb442afb5ede23bd081165a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:36:32 UTC) from {username='oryx', principal='oryx/127.22.37.62@KRBTEST.COM'} at 127.0.0.1:55434
I20260620 18:36:32.095227 30524 tablet_service.cc:1514] Processing DeleteTablet for tablet ce657fac4eb442afb5ede23bd081165a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:36:32 UTC) from {username='oryx', principal='oryx/127.22.37.62@KRBTEST.COM'} at 127.0.0.1:48596
I20260620 18:36:32.096051 30662 tablet_service.cc:1514] Processing DeleteTablet for tablet ce657fac4eb442afb5ede23bd081165a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:36:32 UTC) from {username='oryx', principal='oryx/127.22.37.62@KRBTEST.COM'} at 127.0.0.1:44078
I20260620 18:36:32.096338 30787 tablet_replica.cc:330] T ce657fac4eb442afb5ede23bd081165a P 622c643eaee54366b09419fb11891fe9: stopping tablet replica
I20260620 18:36:32.096508 30787 raft_consensus.cc:2229] T ce657fac4eb442afb5ede23bd081165a P 622c643eaee54366b09419fb11891fe9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:36:32.096767 30787 raft_consensus.cc:2258] T ce657fac4eb442afb5ede23bd081165a P 622c643eaee54366b09419fb11891fe9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:36:32.097105 30248 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43614:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 18:36:32.097613 30789 ts_tablet_manager.cc:1905] T ce657fac4eb442afb5ede23bd081165a P 856016baca2c4cf3b35c8b1c3d8ac989: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 18:36:32.097959 30248 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:36:32.098233 30788 ts_tablet_manager.cc:1918] T ce657fac4eb442afb5ede23bd081165a P b94701dfaaea4c6c95f229005e7b613e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:36:32.098372 30788 log.cc:1196] T ce657fac4eb442afb5ede23bd081165a P b94701dfaaea4c6c95f229005e7b613e: Deleting WAL directory at /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/minicluster-data/ts-0/wal/wals/ce657fac4eb442afb5ede23bd081165a
I20260620 18:36:32.098906 30788 ts_tablet_manager.cc:1939] T ce657fac4eb442afb5ede23bd081165a P b94701dfaaea4c6c95f229005e7b613e: Deleting consensus metadata
I20260620 18:36:32.099934 30235 catalog_manager.cc:4841] TS b94701dfaaea4c6c95f229005e7b613e (127.22.37.1:44231): tablet ce657fac4eb442afb5ede23bd081165a (table test-table [id=aa58a77479034122a8f5813a76ec0fa4]) successfully deleted
W20260620 18:36:32.100826 30235 catalog_manager.cc:4808] TS b94701dfaaea4c6c95f229005e7b613e (127.22.37.1:44231): delete failed for tablet ce657fac4eb442afb5ede23bd081165a because the tablet was not found. No further retry: Not found: Tablet not found: ce657fac4eb442afb5ede23bd081165a
I20260620 18:36:32.105613 30787 ts_tablet_manager.cc:1905] T ce657fac4eb442afb5ede23bd081165a P 622c643eaee54366b09419fb11891fe9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:36:32.105877 30662 tablet_service.cc:1467] Processing CreateTablet for tablet 94c75ecffc0f4ed89556ea62b0b1070d (DEFAULT_TABLE table=test-table [id=5b5a005a3a494801af916d8d7498e6a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:36:32.105877 30524 tablet_service.cc:1467] Processing CreateTablet for tablet 94c75ecffc0f4ed89556ea62b0b1070d (DEFAULT_TABLE table=test-table [id=5b5a005a3a494801af916d8d7498e6a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:36:32.106254 30524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94c75ecffc0f4ed89556ea62b0b1070d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:36:32.106258 30662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94c75ecffc0f4ed89556ea62b0b1070d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:36:32.112390 30790 tablet_bootstrap.cc:492] T 94c75ecffc0f4ed89556ea62b0b1070d P 856016baca2c4cf3b35c8b1c3d8ac989: Bootstrap starting.
I20260620 18:36:32.112361 30787 ts_tablet_manager.cc:1918] T ce657fac4eb442afb5ede23bd081165a P 622c643eaee54366b09419fb11891fe9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:36:32.112506 30787 log.cc:1196] T ce657fac4eb442afb5ede23bd081165a P 622c643eaee54366b09419fb11891fe9: Deleting WAL directory at /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/minicluster-data/ts-2/wal/wals/ce657fac4eb442afb5ede23bd081165a
I20260620 18:36:32.112857 30385 tablet_service.cc:1467] Processing CreateTablet for tablet 94c75ecffc0f4ed89556ea62b0b1070d (DEFAULT_TABLE table=test-table [id=5b5a005a3a494801af916d8d7498e6a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:36:32.112982 30787 ts_tablet_manager.cc:1939] T ce657fac4eb442afb5ede23bd081165a P 622c643eaee54366b09419fb11891fe9: Deleting consensus metadata
I20260620 18:36:32.113265 30385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94c75ecffc0f4ed89556ea62b0b1070d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:36:32.114307 30790 tablet_bootstrap.cc:654] T 94c75ecffc0f4ed89556ea62b0b1070d P 856016baca2c4cf3b35c8b1c3d8ac989: Neither blocks nor log segments found. Creating new log.
I20260620 18:36:32.117862 30791 tablet_bootstrap.cc:492] T 94c75ecffc0f4ed89556ea62b0b1070d P 622c643eaee54366b09419fb11891fe9: Bootstrap starting.
I20260620 18:36:32.119463 30791 tablet_bootstrap.cc:654] T 94c75ecffc0f4ed89556ea62b0b1070d P 622c643eaee54366b09419fb11891fe9: Neither blocks nor log segments found. Creating new log.
I20260620 18:36:32.120639 30235 catalog_manager.cc:4841] TS 622c643eaee54366b09419fb11891fe9 (127.22.37.3:38211): tablet ce657fac4eb442afb5ede23bd081165a (table test-table [id=aa58a77479034122a8f5813a76ec0fa4]) successfully deleted
I20260620 18:36:32.122180 30794 tablet_bootstrap.cc:492] T 94c75ecffc0f4ed89556ea62b0b1070d P b94701dfaaea4c6c95f229005e7b613e: Bootstrap starting.
I20260620 18:36:32.123593 30789 ts_tablet_manager.cc:1918] T ce657fac4eb442afb5ede23bd081165a P 856016baca2c4cf3b35c8b1c3d8ac989: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:36:32.123812 30789 log.cc:1196] T ce657fac4eb442afb5ede23bd081165a P 856016baca2c4cf3b35c8b1c3d8ac989: Deleting WAL directory at /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/minicluster-data/ts-1/wal/wals/ce657fac4eb442afb5ede23bd081165a
I20260620 18:36:32.124234 30794 tablet_bootstrap.cc:654] T 94c75ecffc0f4ed89556ea62b0b1070d P b94701dfaaea4c6c95f229005e7b613e: Neither blocks nor log segments found. Creating new log.
I20260620 18:36:32.124329 30789 ts_tablet_manager.cc:1939] T ce657fac4eb442afb5ede23bd081165a P 856016baca2c4cf3b35c8b1c3d8ac989: Deleting consensus metadata
I20260620 18:36:32.125598 30235 catalog_manager.cc:4841] TS 856016baca2c4cf3b35c8b1c3d8ac989 (127.22.37.2:46153): tablet ce657fac4eb442afb5ede23bd081165a (table test-table [id=aa58a77479034122a8f5813a76ec0fa4]) successfully deleted
W20260620 18:36:32.126061 30235 catalog_manager.cc:4808] TS 856016baca2c4cf3b35c8b1c3d8ac989 (127.22.37.2:46153): delete failed for tablet ce657fac4eb442afb5ede23bd081165a because the tablet was not found. No further retry: Not found: Tablet not found: ce657fac4eb442afb5ede23bd081165a
I20260620 18:36:32.133831 30794 tablet_bootstrap.cc:492] T 94c75ecffc0f4ed89556ea62b0b1070d P b94701dfaaea4c6c95f229005e7b613e: No bootstrap required, opened a new log
I20260620 18:36:32.133991 30794 ts_tablet_manager.cc:1397] T 94c75ecffc0f4ed89556ea62b0b1070d P b94701dfaaea4c6c95f229005e7b613e: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.005s
I20260620 18:36:32.134032 30791 tablet_bootstrap.cc:492] T 94c75ecffc0f4ed89556ea62b0b1070d P 622c643eaee54366b09419fb11891fe9: No bootstrap required, opened a new log
I20260620 18:36:32.134289 30790 tablet_bootstrap.cc:492] T 94c75ecffc0f4ed89556ea62b0b1070d P 856016baca2c4cf3b35c8b1c3d8ac989: No bootstrap required, opened a new log
I20260620 18:36:32.134263 30791 ts_tablet_manager.cc:1397] T 94c75ecffc0f4ed89556ea62b0b1070d P 622c643eaee54366b09419fb11891fe9: Time spent bootstrapping tablet: real 0.016s user 0.013s sys 0.003s
I20260620 18:36:32.134465 30790 ts_tablet_manager.cc:1397] T 94c75ecffc0f4ed89556ea62b0b1070d P 856016baca2c4cf3b35c8b1c3d8ac989: Time spent bootstrapping tablet: real 0.022s user 0.011s sys 0.005s
I20260620 18:36:32.134758 30794 raft_consensus.cc:348] T 94c75ecffc0f4ed89556ea62b0b1070d P b94701dfaaea4c6c95f229005e7b613e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b94701dfaaea4c6c95f229005e7b613e" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44231 } } peers { permanent_uuid: "856016baca2c4cf3b35c8b1c3d8ac989" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 46153 } } peers { permanent_uuid: "622c643eaee54366b09419fb11891fe9" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 38211 } }
I20260620 18:36:32.135017 30794 raft_consensus.cc:374] T 94c75ecffc0f4ed89556ea62b0b1070d P b94701dfaaea4c6c95f229005e7b613e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:36:32.135104 30794 raft_consensus.cc:729] T 94c75ecffc0f4ed89556ea62b0b1070d P b94701dfaaea4c6c95f229005e7b613e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b94701dfaaea4c6c95f229005e7b613e, State: Initialized, Role: FOLLOWER
I20260620 18:36:32.135186 30790 raft_consensus.cc:348] T 94c75ecffc0f4ed89556ea62b0b1070d P 856016baca2c4cf3b35c8b1c3d8ac989 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b94701dfaaea4c6c95f229005e7b613e" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44231 } } peers { permanent_uuid: "856016baca2c4cf3b35c8b1c3d8ac989" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 46153 } } peers { permanent_uuid: "622c643eaee54366b09419fb11891fe9" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 38211 } }
I20260620 18:36:32.135438 30790 raft_consensus.cc:374] T 94c75ecffc0f4ed89556ea62b0b1070d P 856016baca2c4cf3b35c8b1c3d8ac989 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:36:32.135367 30794 consensus_queue.cc:260] T 94c75ecffc0f4ed89556ea62b0b1070d P b94701dfaaea4c6c95f229005e7b613e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b94701dfaaea4c6c95f229005e7b613e" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44231 } } peers { permanent_uuid: "856016baca2c4cf3b35c8b1c3d8ac989" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 46153 } } peers { permanent_uuid: "622c643eaee54366b09419fb11891fe9" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 38211 } }
I20260620 18:36:32.135262 30791 raft_consensus.cc:348] T 94c75ecffc0f4ed89556ea62b0b1070d P 622c643eaee54366b09419fb11891fe9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b94701dfaaea4c6c95f229005e7b613e" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44231 } } peers { permanent_uuid: "856016baca2c4cf3b35c8b1c3d8ac989" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 46153 } } peers { permanent_uuid: "622c643eaee54366b09419fb11891fe9" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 38211 } }
I20260620 18:36:32.135519 30790 raft_consensus.cc:729] T 94c75ecffc0f4ed89556ea62b0b1070d P 856016baca2c4cf3b35c8b1c3d8ac989 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 856016baca2c4cf3b35c8b1c3d8ac989, State: Initialized, Role: FOLLOWER
I20260620 18:36:32.135720 30791 raft_consensus.cc:374] T 94c75ecffc0f4ed89556ea62b0b1070d P 622c643eaee54366b09419fb11891fe9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:36:32.135849 30791 raft_consensus.cc:729] T 94c75ecffc0f4ed89556ea62b0b1070d P 622c643eaee54366b09419fb11891fe9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 622c643eaee54366b09419fb11891fe9, State: Initialized, Role: FOLLOWER
I20260620 18:36:32.135780 30790 consensus_queue.cc:260] T 94c75ecffc0f4ed89556ea62b0b1070d P 856016baca2c4cf3b35c8b1c3d8ac989 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b94701dfaaea4c6c95f229005e7b613e" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44231 } } peers { permanent_uuid: "856016baca2c4cf3b35c8b1c3d8ac989" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 46153 } } peers { permanent_uuid: "622c643eaee54366b09419fb11891fe9" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 38211 } }
I20260620 18:36:32.136317 30790 ts_tablet_manager.cc:1428] T 94c75ecffc0f4ed89556ea62b0b1070d P 856016baca2c4cf3b35c8b1c3d8ac989: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260620 18:36:32.136624 30791 consensus_queue.cc:260] T 94c75ecffc0f4ed89556ea62b0b1070d P 622c643eaee54366b09419fb11891fe9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b94701dfaaea4c6c95f229005e7b613e" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44231 } } peers { permanent_uuid: "856016baca2c4cf3b35c8b1c3d8ac989" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 46153 } } peers { permanent_uuid: "622c643eaee54366b09419fb11891fe9" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 38211 } }
I20260620 18:36:32.138020 30794 ts_tablet_manager.cc:1428] T 94c75ecffc0f4ed89556ea62b0b1070d P b94701dfaaea4c6c95f229005e7b613e: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260620 18:36:32.140117 30791 ts_tablet_manager.cc:1428] T 94c75ecffc0f4ed89556ea62b0b1070d P 622c643eaee54366b09419fb11891fe9: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20260620 18:36:32.370496 30812 raft_consensus.cc:482] T 94c75ecffc0f4ed89556ea62b0b1070d P 622c643eaee54366b09419fb11891fe9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:36:32.370708 30812 raft_consensus.cc:504] T 94c75ecffc0f4ed89556ea62b0b1070d P 622c643eaee54366b09419fb11891fe9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b94701dfaaea4c6c95f229005e7b613e" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44231 } } peers { permanent_uuid: "856016baca2c4cf3b35c8b1c3d8ac989" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 46153 } } peers { permanent_uuid: "622c643eaee54366b09419fb11891fe9" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 38211 } }
I20260620 18:36:32.371335 30812 leader_election.cc:288] T 94c75ecffc0f4ed89556ea62b0b1070d P 622c643eaee54366b09419fb11891fe9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b94701dfaaea4c6c95f229005e7b613e (127.22.37.1:44231), 856016baca2c4cf3b35c8b1c3d8ac989 (127.22.37.2:46153)
I20260620 18:36:32.372057 30409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94c75ecffc0f4ed89556ea62b0b1070d" candidate_uuid: "622c643eaee54366b09419fb11891fe9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b94701dfaaea4c6c95f229005e7b613e" is_pre_election: true
I20260620 18:36:32.372092 30549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94c75ecffc0f4ed89556ea62b0b1070d" candidate_uuid: "622c643eaee54366b09419fb11891fe9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "856016baca2c4cf3b35c8b1c3d8ac989" is_pre_election: true
I20260620 18:36:32.372326 30409 raft_consensus.cc:2454] T 94c75ecffc0f4ed89556ea62b0b1070d P b94701dfaaea4c6c95f229005e7b613e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 622c643eaee54366b09419fb11891fe9 in term 0.
I20260620 18:36:32.372342 30549 raft_consensus.cc:2454] T 94c75ecffc0f4ed89556ea62b0b1070d P 856016baca2c4cf3b35c8b1c3d8ac989 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 622c643eaee54366b09419fb11891fe9 in term 0.
I20260620 18:36:32.372807 30617 leader_election.cc:302] T 94c75ecffc0f4ed89556ea62b0b1070d P 622c643eaee54366b09419fb11891fe9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 622c643eaee54366b09419fb11891fe9, b94701dfaaea4c6c95f229005e7b613e; no voters:
I20260620 18:36:32.373108 30812 raft_consensus.cc:2781] T 94c75ecffc0f4ed89556ea62b0b1070d P 622c643eaee54366b09419fb11891fe9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:36:32.373188 30812 raft_consensus.cc:482] T 94c75ecffc0f4ed89556ea62b0b1070d P 622c643eaee54366b09419fb11891fe9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:36:32.373308 30812 raft_consensus.cc:3037] T 94c75ecffc0f4ed89556ea62b0b1070d P 622c643eaee54366b09419fb11891fe9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:36:32.375288 30812 raft_consensus.cc:504] T 94c75ecffc0f4ed89556ea62b0b1070d P 622c643eaee54366b09419fb11891fe9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b94701dfaaea4c6c95f229005e7b613e" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44231 } } peers { permanent_uuid: "856016baca2c4cf3b35c8b1c3d8ac989" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 46153 } } peers { permanent_uuid: "622c643eaee54366b09419fb11891fe9" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 38211 } }
I20260620 18:36:32.375797 30812 leader_election.cc:288] T 94c75ecffc0f4ed89556ea62b0b1070d P 622c643eaee54366b09419fb11891fe9 [CANDIDATE]: Term 1 election: Requested vote from peers b94701dfaaea4c6c95f229005e7b613e (127.22.37.1:44231), 856016baca2c4cf3b35c8b1c3d8ac989 (127.22.37.2:46153)
I20260620 18:36:32.376305 30409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94c75ecffc0f4ed89556ea62b0b1070d" candidate_uuid: "622c643eaee54366b09419fb11891fe9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b94701dfaaea4c6c95f229005e7b613e"
I20260620 18:36:32.376340 30549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94c75ecffc0f4ed89556ea62b0b1070d" candidate_uuid: "622c643eaee54366b09419fb11891fe9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "856016baca2c4cf3b35c8b1c3d8ac989"
I20260620 18:36:32.376492 30409 raft_consensus.cc:3037] T 94c75ecffc0f4ed89556ea62b0b1070d P b94701dfaaea4c6c95f229005e7b613e [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:36:32.376539 30549 raft_consensus.cc:3037] T 94c75ecffc0f4ed89556ea62b0b1070d P 856016baca2c4cf3b35c8b1c3d8ac989 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:36:32.378747 30409 raft_consensus.cc:2454] T 94c75ecffc0f4ed89556ea62b0b1070d P b94701dfaaea4c6c95f229005e7b613e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 622c643eaee54366b09419fb11891fe9 in term 1.
I20260620 18:36:32.378746 30549 raft_consensus.cc:2454] T 94c75ecffc0f4ed89556ea62b0b1070d P 856016baca2c4cf3b35c8b1c3d8ac989 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 622c643eaee54366b09419fb11891fe9 in term 1.
I20260620 18:36:32.379181 30617 leader_election.cc:302] T 94c75ecffc0f4ed89556ea62b0b1070d P 622c643eaee54366b09419fb11891fe9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 622c643eaee54366b09419fb11891fe9, 856016baca2c4cf3b35c8b1c3d8ac989; no voters:
I20260620 18:36:32.379485 30812 raft_consensus.cc:2781] T 94c75ecffc0f4ed89556ea62b0b1070d P 622c643eaee54366b09419fb11891fe9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:36:32.379848 30812 raft_consensus.cc:686] T 94c75ecffc0f4ed89556ea62b0b1070d P 622c643eaee54366b09419fb11891fe9 [term 1 LEADER]: Becoming Leader. State: Replica: 622c643eaee54366b09419fb11891fe9, State: Running, Role: LEADER
I20260620 18:36:32.380112 30812 consensus_queue.cc:237] T 94c75ecffc0f4ed89556ea62b0b1070d P 622c643eaee54366b09419fb11891fe9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b94701dfaaea4c6c95f229005e7b613e" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44231 } } peers { permanent_uuid: "856016baca2c4cf3b35c8b1c3d8ac989" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 46153 } } peers { permanent_uuid: "622c643eaee54366b09419fb11891fe9" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 38211 } }
I20260620 18:36:32.382629 30246 catalog_manager.cc:5495] T 94c75ecffc0f4ed89556ea62b0b1070d P 622c643eaee54366b09419fb11891fe9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 622c643eaee54366b09419fb11891fe9 (127.22.37.3). New cstate: current_term: 1 leader_uuid: "622c643eaee54366b09419fb11891fe9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b94701dfaaea4c6c95f229005e7b613e" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "856016baca2c4cf3b35c8b1c3d8ac989" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 46153 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "622c643eaee54366b09419fb11891fe9" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 38211 } health_report { overall_health: HEALTHY } } }
I20260620 18:36:32.421875 30857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:32.417528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44094 (local address 127.22.37.3:38211)
0620 18:36:32.418002 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:32.418006 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:32.418021 (+ 15us) server_negotiation.cc:407] Connection header received
0620 18:36:32.418067 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:32.418070 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:32.418125 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:32.418204 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:36:32.418561 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:32.419290 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:32.420456 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:32.420836 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:32.420919 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:32.421280 (+ 361us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:36:32.421396 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:36:32.421454 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:32.421579 (+ 125us) server_negotiation.cc:299] Negotiation successful
0620 18:36:32.421632 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":278,"threads_started":1}
I20260620 18:36:32.434439 30857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:32.429588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.3:35445 (local address 127.22.37.3:38211)
0620 18:36:32.429785 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:32.429794 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:32.430088 (+ 294us) server_negotiation.cc:407] Connection header received
0620 18:36:32.430231 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:32.430234 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:32.430289 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:32.430359 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:36:32.430732 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:32.431411 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:32.432750 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:32.433166 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:32.433308 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:32.433693 (+ 385us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:36:32.433777 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:36:32.433870 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:32.434010 (+ 140us) server_negotiation.cc:299] Negotiation successful
0620 18:36:32.434076 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 18:36:32.434541 30860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:32.429516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.3:38211 (local address 127.22.37.3:35445)
0620 18:36:32.429977 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:32.429993 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:32.430135 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:32.430388 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:32.430391 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:32.430400 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 18:36:32.430601 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:32.430609 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:32.431513 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:32.431516 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:32.432588 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:32.432600 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:32.433323 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:32.433329 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:32.433490 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:32.433532 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 18:36:32.433881 (+ 349us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 18:36:32.433894 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 18:36:32.433993 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 18:36:32.434120 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":248,"threads_started":1}
I20260620 18:36:32.443269 30409 raft_consensus.cc:1261] T 94c75ecffc0f4ed89556ea62b0b1070d P b94701dfaaea4c6c95f229005e7b613e [term 1 FOLLOWER]: Refusing update from remote peer 622c643eaee54366b09419fb11891fe9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:36:32.443293 30549 raft_consensus.cc:1261] T 94c75ecffc0f4ed89556ea62b0b1070d P 856016baca2c4cf3b35c8b1c3d8ac989 [term 1 FOLLOWER]: Refusing update from remote peer 622c643eaee54366b09419fb11891fe9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:36:32.444028 30812 consensus_queue.cc:1035] T 94c75ecffc0f4ed89556ea62b0b1070d P 622c643eaee54366b09419fb11891fe9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b94701dfaaea4c6c95f229005e7b613e" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:36:32.444500 30812 consensus_queue.cc:1035] T 94c75ecffc0f4ed89556ea62b0b1070d P 622c643eaee54366b09419fb11891fe9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "856016baca2c4cf3b35c8b1c3d8ac989" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 46153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:36:32.506613 30246 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43614:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:36:32.506883 30246 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43614:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:36:32.509615 30246 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4d9466ac6e3e4d1c95493d9373c03f45: Sending DeleteTablet for 3 replicas of tablet 94c75ecffc0f4ed89556ea62b0b1070d
I20260620 18:36:32.510380 30385 tablet_service.cc:1514] Processing DeleteTablet for tablet 94c75ecffc0f4ed89556ea62b0b1070d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:36:32 UTC) from {username='oryx', principal='oryx/127.22.37.62@KRBTEST.COM'} at 127.0.0.1:55434
I20260620 18:36:32.510666 30788 tablet_replica.cc:330] T 94c75ecffc0f4ed89556ea62b0b1070d P b94701dfaaea4c6c95f229005e7b613e: stopping tablet replica
I20260620 18:36:32.510814 30524 tablet_service.cc:1514] Processing DeleteTablet for tablet 94c75ecffc0f4ed89556ea62b0b1070d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:36:32 UTC) from {username='oryx', principal='oryx/127.22.37.62@KRBTEST.COM'} at 127.0.0.1:48596
I20260620 18:36:32.510895 30788 raft_consensus.cc:2229] T 94c75ecffc0f4ed89556ea62b0b1070d P b94701dfaaea4c6c95f229005e7b613e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:36:32.511073 30788 raft_consensus.cc:2258] T 94c75ecffc0f4ed89556ea62b0b1070d P b94701dfaaea4c6c95f229005e7b613e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:36:32.511116 30662 tablet_service.cc:1514] Processing DeleteTablet for tablet 94c75ecffc0f4ed89556ea62b0b1070d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:36:32 UTC) from {username='oryx', principal='oryx/127.22.37.62@KRBTEST.COM'} at 127.0.0.1:44078
I20260620 18:36:32.511394 30789 tablet_replica.cc:330] T 94c75ecffc0f4ed89556ea62b0b1070d P 856016baca2c4cf3b35c8b1c3d8ac989: stopping tablet replica
I20260620 18:36:32.511590 30789 raft_consensus.cc:2229] T 94c75ecffc0f4ed89556ea62b0b1070d P 856016baca2c4cf3b35c8b1c3d8ac989 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:36:32.511770 30789 raft_consensus.cc:2258] T 94c75ecffc0f4ed89556ea62b0b1070d P 856016baca2c4cf3b35c8b1c3d8ac989 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:36:32.512034 30787 tablet_replica.cc:330] T 94c75ecffc0f4ed89556ea62b0b1070d P 622c643eaee54366b09419fb11891fe9: stopping tablet replica
I20260620 18:36:32.512243 30788 ts_tablet_manager.cc:1905] T 94c75ecffc0f4ed89556ea62b0b1070d P b94701dfaaea4c6c95f229005e7b613e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:36:32.512234 30787 raft_consensus.cc:2229] T 94c75ecffc0f4ed89556ea62b0b1070d P 622c643eaee54366b09419fb11891fe9 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:36:32.512373 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 30319
I20260620 18:36:32.512734 30787 raft_consensus.cc:2258] T 94c75ecffc0f4ed89556ea62b0b1070d P 622c643eaee54366b09419fb11891fe9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:36:32.515429 30787 ts_tablet_manager.cc:1905] T 94c75ecffc0f4ed89556ea62b0b1070d P 622c643eaee54366b09419fb11891fe9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:36:32.519134 30788 ts_tablet_manager.cc:1918] T 94c75ecffc0f4ed89556ea62b0b1070d P b94701dfaaea4c6c95f229005e7b613e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:36:32.519316 30788 log.cc:1196] T 94c75ecffc0f4ed89556ea62b0b1070d P b94701dfaaea4c6c95f229005e7b613e: Deleting WAL directory at /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/minicluster-data/ts-0/wal/wals/94c75ecffc0f4ed89556ea62b0b1070d
I20260620 18:36:32.519783 30788 ts_tablet_manager.cc:1939] T 94c75ecffc0f4ed89556ea62b0b1070d P b94701dfaaea4c6c95f229005e7b613e: Deleting consensus metadata
I20260620 18:36:32.520856 30235 catalog_manager.cc:4841] TS b94701dfaaea4c6c95f229005e7b613e (127.22.37.1:44231): tablet 94c75ecffc0f4ed89556ea62b0b1070d (table test-table [id=5b5a005a3a494801af916d8d7498e6a3]) successfully deleted
I20260620 18:36:32.522459 30789 ts_tablet_manager.cc:1905] T 94c75ecffc0f4ed89556ea62b0b1070d P 856016baca2c4cf3b35c8b1c3d8ac989: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:36:32.524240 30787 ts_tablet_manager.cc:1918] T 94c75ecffc0f4ed89556ea62b0b1070d P 622c643eaee54366b09419fb11891fe9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:36:32.524397 30787 log.cc:1196] T 94c75ecffc0f4ed89556ea62b0b1070d P 622c643eaee54366b09419fb11891fe9: Deleting WAL directory at /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/minicluster-data/ts-2/wal/wals/94c75ecffc0f4ed89556ea62b0b1070d
I20260620 18:36:32.524824 30787 ts_tablet_manager.cc:1939] T 94c75ecffc0f4ed89556ea62b0b1070d P 622c643eaee54366b09419fb11891fe9: Deleting consensus metadata
Jun 20 18:36:32 dist-test-slave-5lgp krb5kdc[30203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980591, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.37.1@KRBTEST.COM
I20260620 18:36:32.526091 30235 catalog_manager.cc:4841] TS 622c643eaee54366b09419fb11891fe9 (127.22.37.3:38211): tablet 94c75ecffc0f4ed89556ea62b0b1070d (table test-table [id=5b5a005a3a494801af916d8d7498e6a3]) successfully deleted
I20260620 18:36:32.529431 30789 ts_tablet_manager.cc:1918] T 94c75ecffc0f4ed89556ea62b0b1070d P 856016baca2c4cf3b35c8b1c3d8ac989: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:36:32.529582 30789 log.cc:1196] T 94c75ecffc0f4ed89556ea62b0b1070d P 856016baca2c4cf3b35c8b1c3d8ac989: Deleting WAL directory at /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980527431848-22676-0/minicluster-data/ts-1/wal/wals/94c75ecffc0f4ed89556ea62b0b1070d
I20260620 18:36:32.529963 30789 ts_tablet_manager.cc:1939] T 94c75ecffc0f4ed89556ea62b0b1070d P 856016baca2c4cf3b35c8b1c3d8ac989: Deleting consensus metadata
I20260620 18:36:32.531025 30235 catalog_manager.cc:4841] TS 856016baca2c4cf3b35c8b1c3d8ac989 (127.22.37.2:46153): tablet 94c75ecffc0f4ed89556ea62b0b1070d (table test-table [id=5b5a005a3a494801af916d8d7498e6a3]) successfully deleted
I20260620 18:36:32.533233 30848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:32.513389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55452 (local address 127.22.37.1:44231)
0620 18:36:32.513596 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:32.513602 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:32.514168 (+ 566us) server_negotiation.cc:407] Connection header received
0620 18:36:32.514442 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:32.514449 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:32.514531 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:32.514614 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:32.516297 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:32.517562 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:32.519548 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:32.520215 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:32.520308 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:32.526336 (+ 6028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:32.526356 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:32.526378 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:32.526424 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:32.530513 (+ 4089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:32.531377 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:32.531384 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:32.531387 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:32.531498 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:32.531901 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:32.531904 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:32.531906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:32.532257 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:32.532471 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:32.532806 (+ 335us) server_negotiation.cc:299] Negotiation successful
0620 18:36:32.532958 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 18:36:32.534117 30450 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:36:32.656570 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 30319
I20260620 18:36:32.688170 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 30459
Jun 20 18:36:32 dist-test-slave-5lgp krb5kdc[30203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980591, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.37.2@KRBTEST.COM
I20260620 18:36:32.704663 30870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:32.688965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48610 (local address 127.22.37.2:46153)
0620 18:36:32.689477 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:32.689481 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:32.689497 (+ 16us) server_negotiation.cc:407] Connection header received
0620 18:36:32.689552 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:32.689556 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:32.689613 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:32.689720 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:32.691259 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:32.691967 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:32.693753 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:32.694136 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:32.694230 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:32.698990 (+ 4760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:32.699004 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:32.699019 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:32.699053 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:32.701985 (+ 2932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:32.702689 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:32.702697 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:32.702699 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:32.702828 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:32.703189 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:32.703196 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:32.703202 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:32.703548 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:32.703844 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:32.704131 (+ 287us) server_negotiation.cc:299] Negotiation successful
0620 18:36:32.704335 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":260,"threads_started":1}
I20260620 18:36:32.705308 30589 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:36:32.839960 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 30459
I20260620 18:36:32.870889 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 30598
Jun 20 18:36:32 dist-test-slave-5lgp krb5kdc[30203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980591, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.37.3@KRBTEST.COM
I20260620 18:36:32.887614 30857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:32.871761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44104 (local address 127.22.37.3:38211)
0620 18:36:32.871946 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:32.871952 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:32.871976 (+ 24us) server_negotiation.cc:407] Connection header received
0620 18:36:32.872055 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:32.872061 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:32.872138 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:32.872245 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:32.873518 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:32.874754 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:32.875979 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:32.876682 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:32.876837 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:32.880986 (+ 4149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:32.881005 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:32.881028 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:32.881072 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:32.884699 (+ 3627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:32.885316 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:32.885325 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:32.885329 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:32.885737 (+ 408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:32.886115 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:32.886120 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:32.886123 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:32.886520 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:32.886799 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:32.887026 (+ 227us) server_negotiation.cc:299] Negotiation successful
0620 18:36:32.887224 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 18:36:32.888283 30728 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:36:33.048203 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 30598
I20260620 18:36:33.073285 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 30216
I20260620 18:36:33.074522 30288 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:36:33.198200 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 30216
2026-06-20T18:36:33Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6146 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980527431848-22676-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:36:33 dist-test-slave-5lgp krb5kdc[30877](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:36:33 dist-test-slave-5lgp krb5kdc[30877](info): set up 2 sockets
Jun 20 18:36:33 dist-test-slave-5lgp krb5kdc[30877](info): commencing operation
krb5kdc: starting...
W20260620 18:36:34.290372 22676 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.001s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980527431848-22676-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980527431848-22676-0/krb5kdc/test-user.keytab.
Jun 20 18:36:34 dist-test-slave-5lgp krb5kdc[30877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980594, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:36:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:36:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.37.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.37.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.37.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab.
Entry for principal kudu/127.22.37.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab.
I20260620 18:36:34.511041 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980527431848-22676-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980527431848-22676-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980527431848-22676-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.62:34341
--webserver_interface=127.22.37.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab
--principal=kudu/127.22.37.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:33315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.37.62:34341
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:36:34.836200 30890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:36:34.836496 30890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:36:34.836650 30890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:36:34.844000 30890 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:36:34.844105 30890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:36:34.844214 30890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:36:34.844311 30890 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:36:34.844401 30890 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:36:34.854290 30890 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:33315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980527431848-22676-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980527431848-22676-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.37.62:34341
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.62:34341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.37.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980527431848-22676-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:36:34.855566 30890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:36:34.856971 30890 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:36:34.868822 30895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:34.868880 30898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:34.868901 30896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:34.869455 30890 server_base.cc:1034] running on GCE node
I20260620 18:36:34.870273 30890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:36:34.871614 30890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:36:34.872830 30890 hybrid_clock.cc:648] HybridClock initialized: now 1781980594872779 us; error 41 us; skew 500 ppm
I20260620 18:36:34.873155 30890 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-20T18:36:34Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1750 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:36:35 dist-test-slave-5lgp krb5kdc[30904](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:36:35 dist-test-slave-5lgp krb5kdc[30904](info): set up 2 sockets
Jun 20 18:36:35 dist-test-slave-5lgp krb5kdc[30904](info): commencing operation
krb5kdc: starting...
W20260620 18:36:36.046381 22676 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.003s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/krb5kdc/test-user.keytab.
Jun 20 18:36:36 dist-test-slave-5lgp krb5kdc[30904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:36:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:36:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.37.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.37.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.37.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab.
Entry for principal kudu/127.22.37.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab.
I20260620 18:36:36.273263 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.62:40737
--webserver_interface=127.22.37.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab
--principal=kudu/127.22.37.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:44445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.37.62:40737
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:36:36.607301 30917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:36:36.607641 30917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:36:36.607735 30917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:36:36.615159 30917 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:36:36.615300 30917 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:36:36.615357 30917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:36:36.615442 30917 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:36:36.615506 30917 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:36:36.625536 30917 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:44445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.37.62:40737
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab
--principal=kudu/127.22.37.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.62:40737
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.37.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:36:36.626847 30917 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:36:36.628096 30917 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:36:36.640297 30922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:36.640339 30923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:36.640375 30925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:36.641281 30917 server_base.cc:1034] running on GCE node
I20260620 18:36:36.642107 30917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:36:36.643539 30917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:36:36.644807 30917 hybrid_clock.cc:648] HybridClock initialized: now 1781980596644759 us; error 53 us; skew 500 ppm
I20260620 18:36:36.645159 30917 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:36:36 dist-test-slave-5lgp krb5kdc[30904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980596, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:36:36.648173 30917 init.cc:377] Logged in from keytab as kudu/127.22.37.62@KRBTEST.COM (short username kudu)
I20260620 18:36:36.649652 30917 webserver.cc:460] Webserver started at http://127.22.37.62:38029/ using document root <none> and password file <none>
I20260620 18:36:36.650336 30917 fs_manager.cc:359] Metadata directory not provided
I20260620 18:36:36.650484 30917 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:36:36.651055 30917 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:36:36.654273 30917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/master-0/data/instance:
uuid: "5cab226da27e4fd6a95aa1659f293b44"
format_stamp: "Formatted at 2026-06-20 18:36:36 on dist-test-slave-5lgp"
server_key: "e39ac62daeacb78b739a31200b9e1240"
server_key_iv: "0ae8a15e55fcd75404fbad33f8e54db0"
server_key_version: "encryptionkey@0"
I20260620 18:36:36.655028 30917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/master-0/wal/instance:
uuid: "5cab226da27e4fd6a95aa1659f293b44"
format_stamp: "Formatted at 2026-06-20 18:36:36 on dist-test-slave-5lgp"
server_key: "e39ac62daeacb78b739a31200b9e1240"
server_key_iv: "0ae8a15e55fcd75404fbad33f8e54db0"
server_key_version: "encryptionkey@0"
I20260620 18:36:36.660521 30917 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260620 18:36:36.664163 30932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:36.665668 30917 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:36:36.665876 30917 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/master-0/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/master-0/wal
uuid: "5cab226da27e4fd6a95aa1659f293b44"
format_stamp: "Formatted at 2026-06-20 18:36:36 on dist-test-slave-5lgp"
server_key: "e39ac62daeacb78b739a31200b9e1240"
server_key_iv: "0ae8a15e55fcd75404fbad33f8e54db0"
server_key_version: "encryptionkey@0"
I20260620 18:36:36.666085 30917 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:36:36.694705 30917 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:36:36.699136 30917 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:36:36.699448 30917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:36:36.718281 30917 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.62:40737
I20260620 18:36:36.718325 30983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.62:40737 every 8 connection(s)
I20260620 18:36:36.719754 30917 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
I20260620 18:36:36.723505 30984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:36:36.725756 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 30917
I20260620 18:36:36.725968 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/master-0/wal/instance
I20260620 18:36:36.726764 22676 external_mini_cluster.cc:1428] Setting key c9b0ec0784869da159b01b0a21b4386a
I20260620 18:36:36.736290 30984 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cab226da27e4fd6a95aa1659f293b44: Bootstrap starting.
I20260620 18:36:36.740290 30984 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5cab226da27e4fd6a95aa1659f293b44: Neither blocks nor log segments found. Creating new log.
I20260620 18:36:36.741696 30984 log.cc:824] T 00000000000000000000000000000000 P 5cab226da27e4fd6a95aa1659f293b44: Log is configured to *not* fsync() on all Append() calls
Jun 20 18:36:36 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM
I20260620 18:36:36.744869 30984 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cab226da27e4fd6a95aa1659f293b44: No bootstrap required, opened a new log
I20260620 18:36:36.751374 30987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:36.729296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37354 (local address 127.22.37.62:40737)
0620 18:36:36.730323 (+ 1027us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:36.730335 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:36.730378 (+ 43us) server_negotiation.cc:407] Connection header received
0620 18:36:36.732020 (+ 1642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:36.732044 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:36.732481 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:36.732819 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:36.734693 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:36.736051 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:36.737838 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:36.738553 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:36.738687 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:36.743633 (+ 4946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:36.743642 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:36.743658 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:36.743700 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:36.747133 (+ 3433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:36.747915 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:36.747926 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:36.747933 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:36.748093 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:36.748483 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:36.748488 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:36.748491 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:36.749053 (+ 562us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:36.749443 (+ 390us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:36.749948 (+ 505us) server_negotiation.cc:299] Negotiation successful
0620 18:36:36.750356 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":789,"thread_start_us":517,"threads_started":1}
I20260620 18:36:36.753027 30984 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5cab226da27e4fd6a95aa1659f293b44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cab226da27e4fd6a95aa1659f293b44" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 40737 } }
I20260620 18:36:36.753473 30984 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5cab226da27e4fd6a95aa1659f293b44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:36:36.753599 30984 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5cab226da27e4fd6a95aa1659f293b44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cab226da27e4fd6a95aa1659f293b44, State: Initialized, Role: FOLLOWER
I20260620 18:36:36.754257 30984 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5cab226da27e4fd6a95aa1659f293b44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cab226da27e4fd6a95aa1659f293b44" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 40737 } }
I20260620 18:36:36.754534 30984 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5cab226da27e4fd6a95aa1659f293b44 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:36:36.754638 30984 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5cab226da27e4fd6a95aa1659f293b44 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:36:36.754761 30984 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5cab226da27e4fd6a95aa1659f293b44 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:36:36.756924 30984 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5cab226da27e4fd6a95aa1659f293b44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cab226da27e4fd6a95aa1659f293b44" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 40737 } }
I20260620 18:36:36.757577 30984 leader_election.cc:302] T 00000000000000000000000000000000 P 5cab226da27e4fd6a95aa1659f293b44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5cab226da27e4fd6a95aa1659f293b44; no voters:
I20260620 18:36:36.758184 30984 leader_election.cc:288] T 00000000000000000000000000000000 P 5cab226da27e4fd6a95aa1659f293b44 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:36:36.758337 30989 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5cab226da27e4fd6a95aa1659f293b44 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:36:36.759089 30989 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5cab226da27e4fd6a95aa1659f293b44 [term 1 LEADER]: Becoming Leader. State: Replica: 5cab226da27e4fd6a95aa1659f293b44, State: Running, Role: LEADER
I20260620 18:36:36.759619 30989 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5cab226da27e4fd6a95aa1659f293b44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cab226da27e4fd6a95aa1659f293b44" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 40737 } }
I20260620 18:36:36.760370 30984 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5cab226da27e4fd6a95aa1659f293b44 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:36:36.762564 30991 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cab226da27e4fd6a95aa1659f293b44 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5cab226da27e4fd6a95aa1659f293b44. Latest consensus state: current_term: 1 leader_uuid: "5cab226da27e4fd6a95aa1659f293b44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cab226da27e4fd6a95aa1659f293b44" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 40737 } } }
I20260620 18:36:36.763011 30991 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cab226da27e4fd6a95aa1659f293b44 [sys.catalog]: This master's current role is: LEADER
I20260620 18:36:36.770642 30990 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cab226da27e4fd6a95aa1659f293b44 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5cab226da27e4fd6a95aa1659f293b44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cab226da27e4fd6a95aa1659f293b44" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 40737 } } }
I20260620 18:36:36.771008 30990 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cab226da27e4fd6a95aa1659f293b44 [sys.catalog]: This master's current role is: LEADER
I20260620 18:36:36.771335 30996 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:36:36.776186 30996 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:36:36.784030 30996 catalog_manager.cc:1347] Generated new cluster ID: ddeeb02ba2e8412e860cdf8d3dc1ef94
I20260620 18:36:36.784152 30996 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:36:36.801432 30996 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:36:36.803436 30996 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:36:36.816466 30996 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5cab226da27e4fd6a95aa1659f293b44: Generated new TSK 0
I20260620 18:36:36.817488 30996 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.37.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.37.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.37.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.1.keytab.
Entry for principal kudu/127.22.37.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.1.keytab.
I20260620 18:36:36.877487 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.37.1:0
--local_ip_for_outbound_sockets=127.22.37.1
--webserver_interface=127.22.37.1
--webserver_port=0
--tserver_master_addrs=127.22.37.62:40737
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.1.keytab
--principal=kudu/127.22.37.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:44445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:36:37.208245 31010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:36:37.208583 31010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:36:37.208760 31010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:36:37.215968 31010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:36:37.216132 31010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:36:37.216321 31010 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.37.1
I20260620 18:36:37.225904 31010 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:44445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.1.keytab
--principal=kudu/127.22.37.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.37.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.37.1
--webserver_port=0
--tserver_master_addrs=127.22.37.62:40737
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.37.1
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:36:37.227238 31010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:36:37.228538 31010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:36:37.241403 31018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:37.241477 31016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:37.241631 31015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:37.242455 31010 server_base.cc:1034] running on GCE node
I20260620 18:36:37.243083 31010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:36:37.243983 31010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:36:37.245230 31010 hybrid_clock.cc:648] HybridClock initialized: now 1781980597245185 us; error 56 us; skew 500 ppm
I20260620 18:36:37.245599 31010 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:36:37 dist-test-slave-5lgp krb5kdc[30904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980597, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:36:37.248629 31010 init.cc:377] Logged in from keytab as kudu/127.22.37.1@KRBTEST.COM (short username kudu)
I20260620 18:36:37.250200 31010 webserver.cc:460] Webserver started at http://127.22.37.1:33353/ using document root <none> and password file <none>
I20260620 18:36:37.251087 31010 fs_manager.cc:359] Metadata directory not provided
I20260620 18:36:37.251192 31010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:36:37.251442 31010 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:36:37.254338 31010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/ts-0/data/instance:
uuid: "48007885caa34dee851100c0192b948c"
format_stamp: "Formatted at 2026-06-20 18:36:37 on dist-test-slave-5lgp"
server_key: "e76e846bd8d1d76a7481d57046326620"
server_key_iv: "09d43b22501832590f15bf0e76e44e52"
server_key_version: "encryptionkey@0"
I20260620 18:36:37.255043 31010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/ts-0/wal/instance:
uuid: "48007885caa34dee851100c0192b948c"
format_stamp: "Formatted at 2026-06-20 18:36:37 on dist-test-slave-5lgp"
server_key: "e76e846bd8d1d76a7481d57046326620"
server_key_iv: "09d43b22501832590f15bf0e76e44e52"
server_key_version: "encryptionkey@0"
I20260620 18:36:37.260126 31010 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 18:36:37.263585 31025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:37.264973 31010 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 18:36:37.265132 31010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/ts-0/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/ts-0/wal
uuid: "48007885caa34dee851100c0192b948c"
format_stamp: "Formatted at 2026-06-20 18:36:37 on dist-test-slave-5lgp"
server_key: "e76e846bd8d1d76a7481d57046326620"
server_key_iv: "09d43b22501832590f15bf0e76e44e52"
server_key_version: "encryptionkey@0"
I20260620 18:36:37.265324 31010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:36:37.288305 31010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:36:37.292548 31010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:36:37.292838 31010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:36:37.293896 31010 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:36:37.295210 31010 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:36:37.295285 31010 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:37.295454 31010 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:36:37.295538 31010 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:37.328112 31010 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.1:45785
I20260620 18:36:37.328158 31137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.1:45785 every 8 connection(s)
I20260620 18:36:37.329452 31010 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/ts-0/data/info.pb
I20260620 18:36:37.330765 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 31010
I20260620 18:36:37.330965 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/ts-0/wal/instance
I20260620 18:36:37.331722 22676 external_mini_cluster.cc:1428] Setting key cd44ae41f2fbfd405eabff5a6c184c0a
Jun 20 18:36:37 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980597, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.1@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM
I20260620 18:36:37.349718 31140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:37.332187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.1:36665 (local address 127.22.37.62:40737)
0620 18:36:37.332618 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:37.332622 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:37.333905 (+ 1283us) server_negotiation.cc:407] Connection header received
0620 18:36:37.335692 (+ 1787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:37.335697 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:37.335781 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:37.335888 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:37.337289 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:37.338101 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:37.339634 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:37.340050 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:37.340183 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:37.343811 (+ 3628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:37.343816 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:37.343820 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:37.343852 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:37.346474 (+ 2622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:37.347197 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:37.347204 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:37.347208 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:37.347361 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:37.347809 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:37.347817 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:37.347820 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:37.348096 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:37.348269 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:37.349007 (+ 738us) server_negotiation.cc:299] Negotiation successful
0620 18:36:37.349241 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":228,"threads_started":1}
I20260620 18:36:37.350680 31141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:37.332845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:40737 (local address 127.22.37.1:36665)
0620 18:36:37.333736 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:37.333787 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:37.335440 (+ 1653us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:37.336111 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:37.336119 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:37.336602 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:37.337112 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:37.337132 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:37.338251 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:37.338260 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:37.339482 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:37.339493 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:37.340273 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:37.340281 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:37.340512 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:37.341237 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:37.341263 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:37.343577 (+ 2314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:36:37.346618 (+ 3041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:37.346627 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:37.346643 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:37.347061 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:37.347478 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:37.347484 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:37.347488 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:37.347691 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:37.348231 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:36:37.348245 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:36:37.348604 (+ 359us) client_negotiation.cc:769] Sending connection context
0620 18:36:37.348892 (+ 288us) client_negotiation.cc:240] Negotiation successful
0620 18:36:37.349442 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":289,"threads_started":1}
WARNING: no policy specified for kudu/127.22.37.2@KRBTEST.COM; defaulting to no policy
I20260620 18:36:37.352306 31138 heartbeater.cc:347] Connected to a master server at 127.22.37.62:40737
I20260620 18:36:37.352711 31138 heartbeater.cc:464] Registering TS with master...
I20260620 18:36:37.353524 31138 heartbeater.cc:511] Master 127.22.37.62:40737 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.37.2@KRBTEST.COM" created.
I20260620 18:36:37.355747 30949 ts_manager.cc:194] Registered new tserver with Master: 48007885caa34dee851100c0192b948c (127.22.37.1:45785)
I20260620 18:36:37.357189 30949 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.37.1@KRBTEST.COM'} at 127.22.37.1:36665
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.37.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.2.keytab.
Entry for principal kudu/127.22.37.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.2.keytab.
I20260620 18:36:37.380034 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.37.2:0
--local_ip_for_outbound_sockets=127.22.37.2
--webserver_interface=127.22.37.2
--webserver_port=0
--tserver_master_addrs=127.22.37.62:40737
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.2.keytab
--principal=kudu/127.22.37.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:44445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:36:37.701949 31144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:36:37.702217 31144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:36:37.702359 31144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:36:37.709728 31144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:36:37.709832 31144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:36:37.710024 31144 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.37.2
I20260620 18:36:37.719758 31144 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:44445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.2.keytab
--principal=kudu/127.22.37.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.37.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.37.2
--webserver_port=0
--tserver_master_addrs=127.22.37.62:40737
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.37.2
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:36:37.720911 31144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:36:37.722150 31144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:36:37.734978 31150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:37.735047 31149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:37.735020 31152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:37.735877 31144 server_base.cc:1034] running on GCE node
I20260620 18:36:37.736488 31144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:36:37.737571 31144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:36:37.738793 31144 hybrid_clock.cc:648] HybridClock initialized: now 1781980597738755 us; error 48 us; skew 500 ppm
I20260620 18:36:37.739092 31144 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:36:37 dist-test-slave-5lgp krb5kdc[30904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980597, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:36:37.742125 31144 init.cc:377] Logged in from keytab as kudu/127.22.37.2@KRBTEST.COM (short username kudu)
I20260620 18:36:37.743432 31144 webserver.cc:460] Webserver started at http://127.22.37.2:44893/ using document root <none> and password file <none>
I20260620 18:36:37.744079 31144 fs_manager.cc:359] Metadata directory not provided
I20260620 18:36:37.744220 31144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:36:37.744522 31144 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:36:37.747328 31144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/ts-1/data/instance:
uuid: "10b325e501844ff1a11a6d3855f0dd36"
format_stamp: "Formatted at 2026-06-20 18:36:37 on dist-test-slave-5lgp"
server_key: "ec8588ae9019596428e75b7fc94475da"
server_key_iv: "4bc6423b3754f51cecd9721a3feaa780"
server_key_version: "encryptionkey@0"
I20260620 18:36:37.748082 31144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/ts-1/wal/instance:
uuid: "10b325e501844ff1a11a6d3855f0dd36"
format_stamp: "Formatted at 2026-06-20 18:36:37 on dist-test-slave-5lgp"
server_key: "ec8588ae9019596428e75b7fc94475da"
server_key_iv: "4bc6423b3754f51cecd9721a3feaa780"
server_key_version: "encryptionkey@0"
I20260620 18:36:37.752985 31144 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260620 18:36:37.756181 31159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:37.757555 31144 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:36:37.757718 31144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/ts-1/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/ts-1/wal
uuid: "10b325e501844ff1a11a6d3855f0dd36"
format_stamp: "Formatted at 2026-06-20 18:36:37 on dist-test-slave-5lgp"
server_key: "ec8588ae9019596428e75b7fc94475da"
server_key_iv: "4bc6423b3754f51cecd9721a3feaa780"
server_key_version: "encryptionkey@0"
I20260620 18:36:37.757910 31144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:36:37.792168 31144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:36:37.796370 31144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:36:37.796669 31144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:36:37.797680 31144 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:36:37.799010 31144 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:36:37.799111 31144 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:37.799270 31144 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:36:37.799350 31144 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:37.832546 31144 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.2:34745
I20260620 18:36:37.832602 31271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.2:34745 every 8 connection(s)
I20260620 18:36:37.833860 31144 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/ts-1/data/info.pb
I20260620 18:36:37.840744 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 31144
I20260620 18:36:37.840920 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/ts-1/wal/instance
I20260620 18:36:37.841511 22676 external_mini_cluster.cc:1428] Setting key c6afa284ba33734e02cd7155e36e5ff0
Jun 20 18:36:37 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980597, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.2@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM
I20260620 18:36:37.854274 31140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:37.836316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.2:57213 (local address 127.22.37.62:40737)
0620 18:36:37.836505 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:37.836512 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:37.837631 (+ 1119us) server_negotiation.cc:407] Connection header received
0620 18:36:37.839407 (+ 1776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:37.839412 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:37.839501 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:37.839608 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:37.841128 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:37.842167 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:37.843872 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:37.844561 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:37.844696 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:37.848140 (+ 3444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:37.848148 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:37.848156 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:37.848209 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:37.851189 (+ 2980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:37.852073 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:37.852077 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:37.852079 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:37.852200 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:37.852636 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:37.852639 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:37.852641 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:37.852863 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:37.852963 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:37.853854 (+ 891us) server_negotiation.cc:299] Negotiation successful
0620 18:36:37.854013 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 18:36:37.855804 31274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:37.836785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:40737 (local address 127.22.37.2:57213)
0620 18:36:37.837483 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:37.837520 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:37.839140 (+ 1620us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:37.839851 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:37.839859 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:37.840345 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:37.840910 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:37.840934 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:37.842331 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:37.842335 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:37.843725 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:37.843736 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:37.844760 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:37.844765 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:37.844958 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:37.845661 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:37.845685 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:37.847904 (+ 2219us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:36:37.851325 (+ 3421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:37.851335 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:37.851354 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:37.851969 (+ 615us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:37.852306 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:37.852313 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:37.852320 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:37.852538 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:37.852975 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:36:37.852985 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:36:37.853466 (+ 481us) client_negotiation.cc:769] Sending connection context
0620 18:36:37.853778 (+ 312us) client_negotiation.cc:240] Negotiation successful
0620 18:36:37.854362 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":250,"threads_started":1}
I20260620 18:36:37.857262 31272 heartbeater.cc:347] Connected to a master server at 127.22.37.62:40737
I20260620 18:36:37.857590 31272 heartbeater.cc:464] Registering TS with master...
I20260620 18:36:37.858314 31272 heartbeater.cc:511] Master 127.22.37.62:40737 requested a full tablet report, sending...
I20260620 18:36:37.859756 30949 ts_manager.cc:194] Registered new tserver with Master: 10b325e501844ff1a11a6d3855f0dd36 (127.22.37.2:34745)
I20260620 18:36:37.860726 30949 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.37.2@KRBTEST.COM'} at 127.22.37.2:57213
WARNING: no policy specified for kudu/127.22.37.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.37.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.37.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.3.keytab.
Entry for principal kudu/127.22.37.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.3.keytab.
I20260620 18:36:37.892506 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.37.3:0
--local_ip_for_outbound_sockets=127.22.37.3
--webserver_interface=127.22.37.3
--webserver_port=0
--tserver_master_addrs=127.22.37.62:40737
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.3.keytab
--principal=kudu/127.22.37.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:44445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:36:38.217133 31277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:36:38.217456 31277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:36:38.217643 31277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:36:38.224575 31277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:36:38.224665 31277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:36:38.224836 31277 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.37.3
I20260620 18:36:38.234301 31277 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:44445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.3.keytab
--principal=kudu/127.22.37.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.37.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.37.3
--webserver_port=0
--tserver_master_addrs=127.22.37.62:40737
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.37.3
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:36:38.235486 31277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:36:38.236728 31277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:36:38.249774 31282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:38.249688 31283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:38.249783 31285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:38.250417 31277 server_base.cc:1034] running on GCE node
I20260620 18:36:38.251030 31277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:36:38.251935 31277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:36:38.253154 31277 hybrid_clock.cc:648] HybridClock initialized: now 1781980598253118 us; error 63 us; skew 500 ppm
I20260620 18:36:38.253508 31277 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:36:38 dist-test-slave-5lgp krb5kdc[30904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980598, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:36:38.256445 31277 init.cc:377] Logged in from keytab as kudu/127.22.37.3@KRBTEST.COM (short username kudu)
I20260620 18:36:38.257933 31277 webserver.cc:460] Webserver started at http://127.22.37.3:44723/ using document root <none> and password file <none>
I20260620 18:36:38.258558 31277 fs_manager.cc:359] Metadata directory not provided
I20260620 18:36:38.258651 31277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:36:38.258896 31277 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:36:38.261843 31277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/ts-2/data/instance:
uuid: "18e6be55526e492f8af4c93dc42afe0f"
format_stamp: "Formatted at 2026-06-20 18:36:38 on dist-test-slave-5lgp"
server_key: "70fbe53f48b2d6e88960c65b4a59578c"
server_key_iv: "35f299e09ff89c0068554a0a94c8bef6"
server_key_version: "encryptionkey@0"
I20260620 18:36:38.262646 31277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/ts-2/wal/instance:
uuid: "18e6be55526e492f8af4c93dc42afe0f"
format_stamp: "Formatted at 2026-06-20 18:36:38 on dist-test-slave-5lgp"
server_key: "70fbe53f48b2d6e88960c65b4a59578c"
server_key_iv: "35f299e09ff89c0068554a0a94c8bef6"
server_key_version: "encryptionkey@0"
I20260620 18:36:38.267804 31277 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:36:38.271430 31292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:38.272748 31277 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:36:38.272921 31277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/ts-2/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/ts-2/wal
uuid: "18e6be55526e492f8af4c93dc42afe0f"
format_stamp: "Formatted at 2026-06-20 18:36:38 on dist-test-slave-5lgp"
server_key: "70fbe53f48b2d6e88960c65b4a59578c"
server_key_iv: "35f299e09ff89c0068554a0a94c8bef6"
server_key_version: "encryptionkey@0"
I20260620 18:36:38.273111 31277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:36:38.297784 31277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:36:38.301735 31277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:36:38.302028 31277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:36:38.302981 31277 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:36:38.304283 31277 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:36:38.304381 31277 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:38.304528 31277 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:36:38.304597 31277 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:38.336619 31277 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.3:42181
I20260620 18:36:38.336699 31404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.3:42181 every 8 connection(s)
I20260620 18:36:38.337881 31277 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/ts-2/data/info.pb
I20260620 18:36:38.342806 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 31277
I20260620 18:36:38.342983 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/ts-2/wal/instance
I20260620 18:36:38.343415 22676 external_mini_cluster.cc:1428] Setting key 5ad1cf156298fcc2a34aec7160737da6
Jun 20 18:36:38 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980598, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.3@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM
I20260620 18:36:38.356549 31140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:38.340271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.3:50365 (local address 127.22.37.62:40737)
0620 18:36:38.340440 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:38.340444 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:38.341504 (+ 1060us) server_negotiation.cc:407] Connection header received
0620 18:36:38.343355 (+ 1851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:38.343360 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:38.343462 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:38.343587 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:38.345330 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:38.346196 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:38.347664 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:38.348069 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:38.348199 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:38.351337 (+ 3138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:38.351341 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:38.351344 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:38.351374 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:38.353819 (+ 2445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:38.354454 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:38.354459 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:38.354469 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:38.354586 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:38.354972 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:38.354978 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:38.354981 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:38.355216 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:38.355383 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:38.356013 (+ 630us) server_negotiation.cc:299] Negotiation successful
0620 18:36:38.356220 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 18:36:38.357672 31407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:38.340681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:40737 (local address 127.22.37.3:50365)
0620 18:36:38.341331 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:38.341407 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:38.343108 (+ 1701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:38.343876 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:38.343887 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:38.344445 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:38.345173 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:38.345191 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:38.346327 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:38.346332 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:38.347532 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:38.347542 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:38.348282 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:38.348290 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:38.348512 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:38.349099 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:38.349120 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:38.351141 (+ 2021us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:36:38.353942 (+ 2801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:38.353950 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:38.353964 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:38.354339 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:38.354691 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:38.354697 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:38.354700 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:38.354870 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:38.355361 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:36:38.355368 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:36:38.355671 (+ 303us) client_negotiation.cc:769] Sending connection context
0620 18:36:38.355923 (+ 252us) client_negotiation.cc:240] Negotiation successful
0620 18:36:38.356387 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":248,"threads_started":1}
I20260620 18:36:38.359027 31405 heartbeater.cc:347] Connected to a master server at 127.22.37.62:40737
I20260620 18:36:38.359345 31405 heartbeater.cc:464] Registering TS with master...
I20260620 18:36:38.360010 31405 heartbeater.cc:511] Master 127.22.37.62:40737 requested a full tablet report, sending...
I20260620 18:36:38.360384 31138 heartbeater.cc:503] Master 127.22.37.62:40737 was elected leader, sending a full tablet report...
I20260620 18:36:38.361538 30948 ts_manager.cc:194] Registered new tserver with Master: 18e6be55526e492f8af4c93dc42afe0f (127.22.37.3:42181)
I20260620 18:36:38.362432 30948 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.37.3@KRBTEST.COM'} at 127.22.37.3:50365
I20260620 18:36:38.370942 22676 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.22.37.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.37.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.37.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/krb5kdc/oryx_127.22.37.1.keytab.
Entry for principal oryx/127.22.37.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/krb5kdc/oryx_127.22.37.1.keytab.
WARNING: no policy specified for oryx/127.22.37.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.37.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.37.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/krb5kdc/oryx_127.22.37.2.keytab.
Entry for principal oryx/127.22.37.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/krb5kdc/oryx_127.22.37.2.keytab.
WARNING: no policy specified for oryx/127.22.37.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.37.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.37.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/krb5kdc/oryx_127.22.37.3.keytab.
Entry for principal oryx/127.22.37.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/krb5kdc/oryx_127.22.37.3.keytab.
I20260620 18:36:38.503834 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 31010
Jun 20 18:36:38 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.37.1@KRBTEST.COM
I20260620 18:36:38.520251 31415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:38.505083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49346 (local address 127.22.37.1:45785)
0620 18:36:38.505747 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:38.505754 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:38.505781 (+ 27us) server_negotiation.cc:407] Connection header received
0620 18:36:38.505856 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:38.505862 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:38.506077 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:38.506232 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:38.507510 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:38.508530 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:38.509870 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:38.510430 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:38.510528 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:38.514680 (+ 4152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:38.514693 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:38.514708 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:38.514741 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:38.517537 (+ 2796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:38.518241 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:38.518249 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:38.518251 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:38.518379 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:38.518827 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:38.518831 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:38.518833 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:38.519220 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:38.519460 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:38.519731 (+ 271us) server_negotiation.cc:299] Negotiation successful
0620 18:36:38.519952 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":355,"threads_started":1}
I20260620 18:36:38.521600 31133 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:36:38.604092 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 31010
I20260620 18:36:38.620141 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 31144
Jun 20 18:36:38 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.37.2@KRBTEST.COM
I20260620 18:36:38.635752 31417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:38.621061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54076 (local address 127.22.37.2:34745)
0620 18:36:38.621585 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:38.621592 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:38.621616 (+ 24us) server_negotiation.cc:407] Connection header received
0620 18:36:38.621670 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:38.621676 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:38.621892 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:38.622060 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:38.623296 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:38.624277 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:38.625559 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:38.626089 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:38.626229 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:38.630199 (+ 3970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:38.630209 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:38.630222 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:38.630252 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:38.632915 (+ 2663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:38.633692 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:38.633705 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:38.633708 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:38.633833 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:38.634203 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:38.634208 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:38.634210 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:38.634520 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:38.634817 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:38.635145 (+ 328us) server_negotiation.cc:299] Negotiation successful
0620 18:36:38.635401 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":231,"threads_started":1}
I20260620 18:36:38.636982 31267 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:36:38.718860 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 31144
I20260620 18:36:38.735018 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 31277
Jun 20 18:36:38 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.37.3@KRBTEST.COM
I20260620 18:36:38.750504 31419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:38.735846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57788 (local address 127.22.37.3:42181)
0620 18:36:38.736370 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:38.736376 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:38.736423 (+ 47us) server_negotiation.cc:407] Connection header received
0620 18:36:38.736501 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:38.736507 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:38.736844 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:38.737007 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:38.738580 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:38.739581 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:38.740738 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:38.741272 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:38.741445 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:38.745591 (+ 4146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:38.745601 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:38.745616 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:38.745649 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:38.748152 (+ 2503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:38.748751 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:38.748758 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:38.748760 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:38.748874 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:38.749160 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:38.749163 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:38.749165 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:38.749543 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:38.749769 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:38.749999 (+ 230us) server_negotiation.cc:299] Negotiation successful
0620 18:36:38.750196 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":242,"threads_started":1}
I20260620 18:36:38.751787 31400 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:36:38.834358 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 31277
I20260620 18:36:38.850075 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 30917
I20260620 18:36:38.851274 30979 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:36:38.955581 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 30917
I20260620 18:36:38.976379 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.62:40737
--webserver_interface=127.22.37.62
--webserver_port=38029
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab
--principal=kudu/127.22.37.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:44445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.37.62:40737
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:36:39.301803 31422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:36:39.302088 31422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:36:39.302174 31422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:36:39.309325 31422 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:36:39.309484 31422 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:36:39.309538 31422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:36:39.309597 31422 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:36:39.309638 31422 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:36:39.319510 31422 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:44445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.37.62:40737
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab
--principal=kudu/127.22.37.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.62:40737
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.37.62
--webserver_port=38029
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:36:39.320935 31422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:36:39.322284 31422 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:36:39.333588 31428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:39.333691 31427 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:39.333664 31430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:39.334694 31422 server_base.cc:1034] running on GCE node
I20260620 18:36:39.335474 31422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:36:39.336902 31422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:36:39.338128 31422 hybrid_clock.cc:648] HybridClock initialized: now 1781980599338088 us; error 54 us; skew 500 ppm
I20260620 18:36:39.338479 31422 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:36:39 dist-test-slave-5lgp krb5kdc[30904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980599, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:36:39.341490 31422 init.cc:377] Logged in from keytab as kudu/127.22.37.62@KRBTEST.COM (short username kudu)
I20260620 18:36:39.342921 31422 webserver.cc:460] Webserver started at http://127.22.37.62:38029/ using document root <none> and password file <none>
I20260620 18:36:39.343627 31422 fs_manager.cc:359] Metadata directory not provided
I20260620 18:36:39.343782 31422 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:36:39.349327 31422 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 18:36:39.352315 31437 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:39.353789 31422 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:36:39.353964 31422 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/master-0/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/master-0/wal
uuid: "5cab226da27e4fd6a95aa1659f293b44"
format_stamp: "Formatted at 2026-06-20 18:36:36 on dist-test-slave-5lgp"
server_key: "e39ac62daeacb78b739a31200b9e1240"
server_key_iv: "0ae8a15e55fcd75404fbad33f8e54db0"
server_key_version: "encryptionkey@0"
I20260620 18:36:39.354943 31422 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:36:39.371526 31422 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:36:39.375293 31422 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:36:39.375560 31422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:36:39.393219 31422 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.62:40737
I20260620 18:36:39.393194 31488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.62:40737 every 8 connection(s)
I20260620 18:36:39.394712 31422 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
I20260620 18:36:39.397249 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 31422
I20260620 18:36:39.398085 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.37.1:45785
--local_ip_for_outbound_sockets=127.22.37.1
--tserver_master_addrs=127.22.37.62:40737
--webserver_port=33353
--webserver_interface=127.22.37.1
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.1.keytab
--principal=kudu/127.22.37.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:44445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.22.37.1
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/krb5kdc/oryx_127.22.37.1.keytab with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:36:39.401618 31489 sys_catalog.cc:263] Verifying existing consensus state
I20260620 18:36:39.404927 31489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cab226da27e4fd6a95aa1659f293b44: Bootstrap starting.
I20260620 18:36:39.432255 31489 log.cc:824] T 00000000000000000000000000000000 P 5cab226da27e4fd6a95aa1659f293b44: Log is configured to *not* fsync() on all Append() calls
I20260620 18:36:39.443045 31489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cab226da27e4fd6a95aa1659f293b44: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 18:36:39.443642 31489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cab226da27e4fd6a95aa1659f293b44: Bootstrap complete.
I20260620 18:36:39.457057 31489 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5cab226da27e4fd6a95aa1659f293b44 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cab226da27e4fd6a95aa1659f293b44" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 40737 } }
I20260620 18:36:39.458323 31489 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5cab226da27e4fd6a95aa1659f293b44 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cab226da27e4fd6a95aa1659f293b44, State: Initialized, Role: FOLLOWER
I20260620 18:36:39.459039 31489 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5cab226da27e4fd6a95aa1659f293b44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cab226da27e4fd6a95aa1659f293b44" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 40737 } }
I20260620 18:36:39.459306 31489 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5cab226da27e4fd6a95aa1659f293b44 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:36:39.459424 31489 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5cab226da27e4fd6a95aa1659f293b44 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:36:39.459622 31489 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5cab226da27e4fd6a95aa1659f293b44 [term 1 FOLLOWER]: Advancing to term 2
I20260620 18:36:39.466071 31489 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5cab226da27e4fd6a95aa1659f293b44 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cab226da27e4fd6a95aa1659f293b44" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 40737 } }
I20260620 18:36:39.466708 31489 leader_election.cc:302] T 00000000000000000000000000000000 P 5cab226da27e4fd6a95aa1659f293b44 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5cab226da27e4fd6a95aa1659f293b44; no voters:
I20260620 18:36:39.467366 31489 leader_election.cc:288] T 00000000000000000000000000000000 P 5cab226da27e4fd6a95aa1659f293b44 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 18:36:39.467551 31493 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5cab226da27e4fd6a95aa1659f293b44 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 18:36:39.468878 31493 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5cab226da27e4fd6a95aa1659f293b44 [term 2 LEADER]: Becoming Leader. State: Replica: 5cab226da27e4fd6a95aa1659f293b44, State: Running, Role: LEADER
I20260620 18:36:39.469412 31489 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5cab226da27e4fd6a95aa1659f293b44 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:36:39.469462 31493 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5cab226da27e4fd6a95aa1659f293b44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cab226da27e4fd6a95aa1659f293b44" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 40737 } }
I20260620 18:36:39.472012 31494 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cab226da27e4fd6a95aa1659f293b44 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "5cab226da27e4fd6a95aa1659f293b44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cab226da27e4fd6a95aa1659f293b44" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 40737 } } }
I20260620 18:36:39.472012 31495 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cab226da27e4fd6a95aa1659f293b44 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5cab226da27e4fd6a95aa1659f293b44. Latest consensus state: current_term: 2 leader_uuid: "5cab226da27e4fd6a95aa1659f293b44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cab226da27e4fd6a95aa1659f293b44" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 40737 } } }
I20260620 18:36:39.472441 31494 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cab226da27e4fd6a95aa1659f293b44 [sys.catalog]: This master's current role is: LEADER
I20260620 18:36:39.472504 31495 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cab226da27e4fd6a95aa1659f293b44 [sys.catalog]: This master's current role is: LEADER
I20260620 18:36:39.484923 31499 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:36:39.491060 31499 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:36:39.492455 31499 catalog_manager.cc:1259] Loaded cluster ID: ddeeb02ba2e8412e860cdf8d3dc1ef94
I20260620 18:36:39.492609 31499 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:36:39.496487 31499 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:36:39.498050 31499 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5cab226da27e4fd6a95aa1659f293b44: Loaded TSK: 0
I20260620 18:36:39.499569 31499 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 18:36:39.767566 31491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:36:39.767881 31491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:36:39.768051 31491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:36:39.775755 31491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:36:39.775883 31491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:36:39.776072 31491 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.37.1
I20260620 18:36:39.786418 31491 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:44445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/krb5kdc/oryx_127.22.37.1.keytab
--principal=oryx/127.22.37.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.37.1:45785
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.37.1
--webserver_port=33353
--tserver_master_addrs=127.22.37.62:40737
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.37.1
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:36:39.787907 31491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:36:39.789274 31491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:36:39.802999 31518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:39.802994 31516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:39.803767 31515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:39.804358 31491 server_base.cc:1034] running on GCE node
I20260620 18:36:39.805076 31491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:36:39.806236 31491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:36:39.807487 31491 hybrid_clock.cc:648] HybridClock initialized: now 1781980599807433 us; error 41 us; skew 500 ppm
I20260620 18:36:39.807883 31491 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:36:39 dist-test-slave-5lgp krb5kdc[30904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980599, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.37.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:36:39.812014 31491 init.cc:377] Logged in from keytab as oryx/127.22.37.1@KRBTEST.COM (short username oryx)
I20260620 18:36:39.813760 31491 webserver.cc:460] Webserver started at http://127.22.37.1:33353/ using document root <none> and password file <none>
I20260620 18:36:39.814623 31491 fs_manager.cc:359] Metadata directory not provided
I20260620 18:36:39.814761 31491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:36:39.821811 31491 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.002s
I20260620 18:36:39.825039 31525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:39.826704 31491 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:36:39.826860 31491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/ts-0/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/ts-0/wal
uuid: "48007885caa34dee851100c0192b948c"
format_stamp: "Formatted at 2026-06-20 18:36:37 on dist-test-slave-5lgp"
server_key: "e76e846bd8d1d76a7481d57046326620"
server_key_iv: "09d43b22501832590f15bf0e76e44e52"
server_key_version: "encryptionkey@0"
I20260620 18:36:39.828145 31491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:36:39.849599 31491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:36:39.853425 31491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:36:39.853658 31491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:36:39.854629 31491 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:36:39.855924 31491 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:36:39.856012 31491 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:39.856084 31491 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:36:39.856138 31491 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:39.890224 31491 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.1:45785
I20260620 18:36:39.890308 31637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.1:45785 every 8 connection(s)
I20260620 18:36:39.891741 31491 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/ts-0/data/info.pb
I20260620 18:36:39.893446 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 31491
I20260620 18:36:39.894210 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.37.2:34745
--local_ip_for_outbound_sockets=127.22.37.2
--tserver_master_addrs=127.22.37.62:40737
--webserver_port=44893
--webserver_interface=127.22.37.2
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.2.keytab
--principal=kudu/127.22.37.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:44445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.22.37.2
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/krb5kdc/oryx_127.22.37.2.keytab with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 18:36:39 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.37.1@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:36:39 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.37.1@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:36:39.915098 31641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:39.895276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:40737 (local address 127.22.37.1:49575)
0620 18:36:39.896127 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:39.896168 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:39.897925 (+ 1757us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:39.899351 (+ 1426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:39.899363 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:39.899992 (+ 629us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:39.900537 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:39.900555 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:39.902327 (+ 1772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:39.902331 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:39.903609 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:39.903621 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:39.904728 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:39.904736 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:39.905002 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:39.909651 (+ 4649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:39.909683 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:39.913716 (+ 4033us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.37.62:40737: Server oryx/127.22.37.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":414,"threads_started":1}
W20260620 18:36:39.915130 31640 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:36:39.895110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.1:49575 (local address 127.22.37.62:40737)
0620 18:36:39.895812 (+ 702us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:39.895830 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:39.896306 (+ 476us) server_negotiation.cc:407] Connection header received
0620 18:36:39.898486 (+ 2180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:39.898500 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:39.898836 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:39.899193 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:39.900746 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:39.902145 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:39.903798 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:39.904468 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:39.904669 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:39.914122 (+ 9453us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.37.1:49575: BlockingRecv error: recv got EOF from 127.22.37.1:49575 (error 108)
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":253,"threads_started":1}
W20260620 18:36:39.916079 31638 heartbeater.cc:644] Failed to heartbeat to 127.22.37.62:40737 (0 consecutive failures): Not authorized: Failed to ping master at 127.22.37.62:40737: Client connection negotiation failed: client connection to 127.22.37.62:40737: Server oryx/127.22.37.62@KRBTEST.COM not found in Kerberos database
Jun 20 18:36:39 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.37.1@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:36:39 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.37.1@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:36:39.925417 31641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:39.916685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:40737 (local address 127.22.37.1:51693)
0620 18:36:39.916881 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:39.916900 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:39.917007 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:39.917302 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:39.917307 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:39.917710 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:39.917898 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:39.917909 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:39.918927 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:39.918932 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:39.920386 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:39.920401 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:39.921058 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:39.921064 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:39.921242 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:39.921848 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:39.921871 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:39.925128 (+ 3257us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.37.62:40737: Server oryx/127.22.37.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":58}
W20260620 18:36:39.925734 31640 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:36:39.916898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.1:51693 (local address 127.22.37.62:40737)
0620 18:36:39.917046 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:39.917050 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:39.917068 (+ 18us) server_negotiation.cc:407] Connection header received
0620 18:36:39.917114 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:39.917118 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:39.917172 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:39.917272 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:39.918032 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:39.918801 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:39.920528 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:39.920924 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:39.921017 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:39.925317 (+ 4300us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.37.1:51693: BlockingRecv error: recv got EOF from 127.22.37.1:51693 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 20 18:36:39 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.37.1@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:36:39 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.37.1@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:36:39.934221 31641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:39.926079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:40737 (local address 127.22.37.1:44277)
0620 18:36:39.926248 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:39.926266 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:39.926357 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:39.926772 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:39.926777 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:39.927128 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:39.927302 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:39.927312 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:39.928250 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:39.928255 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:39.929580 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:39.929594 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:39.930240 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:39.930245 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:39.930427 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:39.930955 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:39.930975 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:39.933990 (+ 3015us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.37.62:40737: Server oryx/127.22.37.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 18:36:39.934458 31640 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:36:39.926156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.1:44277 (local address 127.22.37.62:40737)
0620 18:36:39.926406 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:39.926414 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:39.926438 (+ 24us) server_negotiation.cc:407] Connection header received
0620 18:36:39.926499 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:39.926506 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:39.926618 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:39.926735 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:39.927431 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:39.928109 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:39.929704 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:39.930105 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:39.930196 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:39.934194 (+ 3998us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.37.1:44277: BlockingRecv error: recv got EOF from 127.22.37.1:44277 (error 108)
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":876,"threads_started":1}
W20260620 18:36:39.934592 31638 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 18:36:40.220803 31642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:36:40.221151 31642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:36:40.221338 31642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:36:40.228528 31642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:36:40.228680 31642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:36:40.228868 31642 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.37.2
I20260620 18:36:40.238775 31642 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:44445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/krb5kdc/oryx_127.22.37.2.keytab
--principal=oryx/127.22.37.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.37.2:34745
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.37.2
--webserver_port=44893
--tserver_master_addrs=127.22.37.62:40737
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.37.2
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:36:40.240155 31642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:36:40.241554 31642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:36:40.254509 31651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:40.254565 31649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:40.254617 31648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:40.255470 31642 server_base.cc:1034] running on GCE node
I20260620 18:36:40.256094 31642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:36:40.257153 31642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:36:40.258399 31642 hybrid_clock.cc:648] HybridClock initialized: now 1781980600258330 us; error 56 us; skew 500 ppm
I20260620 18:36:40.258713 31642 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:36:40 dist-test-slave-5lgp krb5kdc[30904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980600, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.37.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:36:40.261861 31642 init.cc:377] Logged in from keytab as oryx/127.22.37.2@KRBTEST.COM (short username oryx)
I20260620 18:36:40.263325 31642 webserver.cc:460] Webserver started at http://127.22.37.2:44893/ using document root <none> and password file <none>
I20260620 18:36:40.264081 31642 fs_manager.cc:359] Metadata directory not provided
I20260620 18:36:40.264240 31642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:36:40.270174 31642 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 18:36:40.272740 31658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:40.274179 31642 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 18:36:40.274363 31642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/ts-1/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/ts-1/wal
uuid: "10b325e501844ff1a11a6d3855f0dd36"
format_stamp: "Formatted at 2026-06-20 18:36:37 on dist-test-slave-5lgp"
server_key: "ec8588ae9019596428e75b7fc94475da"
server_key_iv: "4bc6423b3754f51cecd9721a3feaa780"
server_key_version: "encryptionkey@0"
I20260620 18:36:40.275315 31642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:36:40.311035 31642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:36:40.315204 31642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:36:40.315491 31642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:36:40.316485 31642 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:36:40.317862 31642 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:36:40.317996 31642 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:40.318118 31642 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:36:40.318204 31642 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:40.350872 31642 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.2:34745
I20260620 18:36:40.350956 31770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.2:34745 every 8 connection(s)
I20260620 18:36:40.352290 31642 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/ts-1/data/info.pb
I20260620 18:36:40.356393 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 31642
I20260620 18:36:40.357012 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.37.3:42181
--local_ip_for_outbound_sockets=127.22.37.3
--tserver_master_addrs=127.22.37.62:40737
--webserver_port=44723
--webserver_interface=127.22.37.3
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.3.keytab
--principal=kudu/127.22.37.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:44445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.22.37.3
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/krb5kdc/oryx_127.22.37.3.keytab with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 18:36:40 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.37.2@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:36:40 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.37.2@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 18:36:40.368301 31640 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:36:40.354803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.2:60533 (local address 127.22.37.62:40737)
0620 18:36:40.354976 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:40.354980 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:40.356151 (+ 1171us) server_negotiation.cc:407] Connection header received
0620 18:36:40.358287 (+ 2136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:40.358293 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:40.358380 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:40.358502 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:40.360006 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:40.360702 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:40.362312 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:40.362729 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:40.362886 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:40.368006 (+ 5120us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.37.2:60533: BlockingRecv error: recv got EOF from 127.22.37.2:60533 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 18:36:40.368901 31773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:40.355299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:40737 (local address 127.22.37.2:60533)
0620 18:36:40.355990 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:40.356029 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:40.358018 (+ 1989us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:40.358775 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:40.358784 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:40.359278 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:40.359824 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:40.359844 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:40.360892 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:40.360898 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:40.362170 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:40.362182 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:40.362957 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:40.362965 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:40.363189 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:40.363855 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:40.363894 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:40.367764 (+ 3870us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.37.62:40737: Server oryx/127.22.37.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":265,"threads_started":1}
W20260620 18:36:40.369645 31771 heartbeater.cc:644] Failed to heartbeat to 127.22.37.62:40737 (0 consecutive failures): Not authorized: Failed to ping master at 127.22.37.62:40737: Client connection negotiation failed: client connection to 127.22.37.62:40737: Server oryx/127.22.37.62@KRBTEST.COM not found in Kerberos database
Jun 20 18:36:40 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.37.2@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:36:40 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.37.2@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:36:40.378304 31773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:40.370222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:40737 (local address 127.22.37.2:57741)
0620 18:36:40.370400 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:40.370420 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:40.370509 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:40.370872 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:40.370875 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:40.371215 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:40.371338 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:40.371346 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:40.372765 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:40.372768 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:40.373748 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:40.373759 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:40.374786 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:40.374790 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:40.374912 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:40.375467 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:40.375483 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:40.378104 (+ 2621us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.37.62:40737: Server oryx/127.22.37.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":33}
W20260620 18:36:40.378778 31640 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:36:40.370375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.2:57741 (local address 127.22.37.62:40737)
0620 18:36:40.370563 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:40.370571 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:40.370590 (+ 19us) server_negotiation.cc:407] Connection header received
0620 18:36:40.370640 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:40.370648 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:40.370740 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:40.370846 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:40.371470 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:40.372643 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:40.373877 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:40.374641 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:40.374779 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:40.378354 (+ 3575us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.37.2:57741: BlockingRecv error: recv got EOF from 127.22.37.2:57741 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 20 18:36:40 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.37.2@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:36:40 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.37.2@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:36:40.387008 31773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:40.378989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:40737 (local address 127.22.37.2:39613)
0620 18:36:40.379152 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:40.379172 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:40.379267 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:40.379764 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:40.379767 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:40.380038 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:40.380169 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:40.380175 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:40.381547 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:40.381551 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:40.382467 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:40.382477 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:40.383339 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:40.383343 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:40.383465 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:40.384021 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:40.384036 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:40.386744 (+ 2708us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.37.62:40737: Server oryx/127.22.37.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":58}
W20260620 18:36:40.387048 31640 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:36:40.379346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.2:39613 (local address 127.22.37.62:40737)
0620 18:36:40.379508 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:40.379513 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:40.379529 (+ 16us) server_negotiation.cc:407] Connection header received
0620 18:36:40.379573 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:40.379578 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:40.379639 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:40.379730 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:40.380319 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:40.381427 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:40.382585 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:40.383205 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:40.383319 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:40.386905 (+ 3586us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.37.2:39613: BlockingRecv error: recv got EOF from 127.22.37.2:39613 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260620 18:36:40.387266 31771 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 18:36:40.684662 31774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:36:40.684994 31774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:36:40.685163 31774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:36:40.692553 31774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:36:40.692704 31774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:36:40.692895 31774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.37.3
I20260620 18:36:40.702575 31774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:44445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/krb5kdc/oryx_127.22.37.3.keytab
--principal=oryx/127.22.37.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.37.3:42181
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.37.3
--webserver_port=44723
--tserver_master_addrs=127.22.37.62:40737
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.37.3
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:36:40.703912 31774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:36:40.705206 31774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:36:40.718489 31780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:40.718454 31779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:40.719019 31782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:40.719370 31774 server_base.cc:1034] running on GCE node
I20260620 18:36:40.719960 31774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:36:40.720902 31774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:36:40.722136 31774 hybrid_clock.cc:648] HybridClock initialized: now 1781980600722089 us; error 38 us; skew 500 ppm
I20260620 18:36:40.722492 31774 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:36:40 dist-test-slave-5lgp krb5kdc[30904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980600, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.37.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:36:40.725618 31774 init.cc:377] Logged in from keytab as oryx/127.22.37.3@KRBTEST.COM (short username oryx)
I20260620 18:36:40.727099 31774 webserver.cc:460] Webserver started at http://127.22.37.3:44723/ using document root <none> and password file <none>
I20260620 18:36:40.727780 31774 fs_manager.cc:359] Metadata directory not provided
I20260620 18:36:40.727943 31774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:36:40.733306 31774 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:36:40.736074 31789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:40.737874 31774 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:36:40.738050 31774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/ts-2/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/ts-2/wal
uuid: "18e6be55526e492f8af4c93dc42afe0f"
format_stamp: "Formatted at 2026-06-20 18:36:38 on dist-test-slave-5lgp"
server_key: "70fbe53f48b2d6e88960c65b4a59578c"
server_key_iv: "35f299e09ff89c0068554a0a94c8bef6"
server_key_version: "encryptionkey@0"
I20260620 18:36:40.739131 31774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:36:40.755318 31774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:36:40.759465 31774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:36:40.759753 31774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:36:40.760711 31774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:36:40.762032 31774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:36:40.762158 31774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:40.762290 31774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:36:40.762388 31774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:40.797744 31774 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.3:42181
I20260620 18:36:40.797780 31901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.3:42181 every 8 connection(s)
I20260620 18:36:40.799523 31774 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980527431848-22676-0/minicluster-data/ts-2/data/info.pb
I20260620 18:36:40.799883 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 31774
Jun 20 18:36:40 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.37.3@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:36:40 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.37.3@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 18:36:40.818336 31905 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:36:40.802499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.3:46371 (local address 127.22.37.62:40737)
0620 18:36:40.803034 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:40.803043 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:40.804190 (+ 1147us) server_negotiation.cc:407] Connection header received
0620 18:36:40.807077 (+ 2887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:40.807082 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:40.807159 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:40.807280 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:40.809346 (+ 2066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:40.810300 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:40.812432 (+ 2132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:40.812858 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:40.812959 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:40.818055 (+ 5096us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.37.3:46371: BlockingRecv error: recv got EOF from 127.22.37.3:46371 (error 108)
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":259,"threads_started":1}
I20260620 18:36:40.819010 31906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:40.802985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:40737 (local address 127.22.37.3:46371)
0620 18:36:40.803981 (+ 996us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:40.804043 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:40.806810 (+ 2767us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:40.807576 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:40.807591 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:40.808372 (+ 781us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:40.809170 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:40.809190 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:40.810451 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:40.810455 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:40.812259 (+ 1804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:40.812276 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:40.813087 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:40.813095 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:40.813439 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:40.814200 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:40.814221 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:40.817891 (+ 3670us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.37.62:40737: Server oryx/127.22.37.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":334,"threads_started":1}
W20260620 18:36:40.819828 31902 heartbeater.cc:644] Failed to heartbeat to 127.22.37.62:40737 (0 consecutive failures): Not authorized: Failed to ping master at 127.22.37.62:40737: Client connection negotiation failed: client connection to 127.22.37.62:40737: Server oryx/127.22.37.62@KRBTEST.COM not found in Kerberos database
I20260620 18:36:40.820802 31640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:40.801294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46188 (local address 127.22.37.62:40737)
0620 18:36:40.801545 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:40.801555 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:40.805913 (+ 4358us) server_negotiation.cc:407] Connection header received
0620 18:36:40.806049 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:40.806055 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:40.806144 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:40.806268 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:40.807631 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:40.808813 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:40.810097 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:40.810695 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:40.810796 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:40.814047 (+ 3251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:40.814059 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:40.814081 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:40.814144 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:40.817479 (+ 3335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:40.818292 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:40.818301 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:40.818309 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:40.818524 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:40.818920 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:40.818925 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:40.818928 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:40.819410 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:40.819757 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:40.820154 (+ 397us) server_negotiation.cc:299] Negotiation successful
0620 18:36:40.820476 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
Jun 20 18:36:40 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.37.3@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:36:40 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.37.3@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:36:40.828809 31906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:40.820455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:40737 (local address 127.22.37.3:49441)
0620 18:36:40.820615 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:40.820635 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:40.820723 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:40.821254 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:40.821258 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:40.821649 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:40.821845 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:40.821857 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:40.823855 (+ 1998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:40.823860 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:40.824721 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:40.824731 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:40.825400 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:40.825405 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:40.825575 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:40.826017 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:40.826032 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:40.828546 (+ 2514us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.37.62:40737: Server oryx/127.22.37.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 18:36:40.828934 31905 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:36:40.820802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.3:49441 (local address 127.22.37.62:40737)
0620 18:36:40.820986 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:40.820991 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:40.821005 (+ 14us) server_negotiation.cc:407] Connection header received
0620 18:36:40.821045 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:40.821049 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:40.821112 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:40.821199 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:40.822772 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:40.823713 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:40.824853 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:40.825242 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:40.825396 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:40.828718 (+ 3322us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.37.3:49441: BlockingRecv error: recv got EOF from 127.22.37.3:49441 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Jun 20 18:36:40 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.37.3@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:36:40 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.37.3@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:36:40.836815 31906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:40.829509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:40737 (local address 127.22.37.3:39653)
0620 18:36:40.829631 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:40.829644 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:40.829741 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:40.830076 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:40.830080 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:40.830347 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:40.830475 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:40.830482 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:40.831415 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:40.831421 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:40.832292 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:40.832301 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:40.832988 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:40.832994 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:40.833170 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:40.833788 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:40.833804 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:40.836624 (+ 2820us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.37.62:40737: Server oryx/127.22.37.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":35}
W20260620 18:36:40.837113 31902 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 18:36:40.837219 31905 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:36:40.829596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.3:39653 (local address 127.22.37.62:40737)
0620 18:36:40.829852 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:40.829857 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:40.829872 (+ 15us) server_negotiation.cc:407] Connection header received
0620 18:36:40.829915 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:40.829918 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:40.829963 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:40.830056 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:40.830601 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:40.831267 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:40.832425 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:40.832860 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:40.832970 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:40.836801 (+ 3831us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.37.3:39653: BlockingRecv error: recv got EOF from 127.22.37.3:39653 (error 108)
Metrics: {"server-negotiator.queue_time_us":140}
Jun 20 18:36:40 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.37.1@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:36:40 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.37.1@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:36:40.943528 31907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:40.935297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:40737 (local address 127.22.37.1:57661)
0620 18:36:40.935779 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:40.935795 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:40.935909 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:40.936195 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:40.936198 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:40.936647 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:40.936832 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:40.936842 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:40.937874 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:40.937879 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:40.938856 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:40.938869 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:40.939567 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:40.939572 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:40.939691 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:40.940261 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:40.940279 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:40.943277 (+ 2998us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.37.62:40737: Server oryx/127.22.37.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":256,"threads_started":1}
W20260620 18:36:40.943751 31905 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:36:40.935358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.1:57661 (local address 127.22.37.62:40737)
0620 18:36:40.935586 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:40.935594 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:40.935869 (+ 275us) server_negotiation.cc:407] Connection header received
0620 18:36:40.936001 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:40.936004 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:40.936075 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:40.936154 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:40.936951 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:40.937711 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:40.939008 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:40.939406 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:40.939522 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:40.943500 (+ 3978us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.37.1:57661: BlockingRecv error: recv got EOF from 127.22.37.1:57661 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 20 18:36:41 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.37.2@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:36:41 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.37.2@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:36:41.397109 31908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:41.387878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:40737 (local address 127.22.37.2:38349)
0620 18:36:41.388399 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:41.388421 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:41.388549 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:41.388879 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:41.388882 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:41.389358 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:41.389612 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:41.389621 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:41.390649 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:41.390654 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:41.391741 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:41.391753 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:41.392827 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:41.392834 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:41.393013 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:41.393749 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:41.393766 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:41.396762 (+ 2996us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.37.62:40737: Server oryx/127.22.37.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":277,"threads_started":1}
W20260620 18:36:41.397228 31905 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:36:41.388066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.2:38349 (local address 127.22.37.62:40737)
0620 18:36:41.388293 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:41.388299 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:41.388504 (+ 205us) server_negotiation.cc:407] Connection header received
0620 18:36:41.388655 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:41.388659 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:41.388719 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:41.388800 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:41.389767 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:41.390506 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:41.391966 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:41.392647 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:41.392790 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:41.396974 (+ 4184us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.37.2:38349: BlockingRecv error: recv got EOF from 127.22.37.2:38349 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Jun 20 18:36:41 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.37.3@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:36:41 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.37.3@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:36:41.845988 31909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:41.837838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:40737 (local address 127.22.37.3:42519)
0620 18:36:41.838320 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:41.838335 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:41.838500 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:41.838892 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:41.838895 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:41.839270 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:41.839426 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:41.839434 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:41.840358 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:41.840361 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:41.841295 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:41.841305 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:41.842084 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:41.842089 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:41.842235 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:41.842947 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:41.842965 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:41.845758 (+ 2793us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.37.62:40737: Server oryx/127.22.37.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":248,"threads_started":1}
W20260620 18:36:41.846163 31905 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:36:41.837873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.3:42519 (local address 127.22.37.62:40737)
0620 18:36:41.838038 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:41.838044 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:41.838419 (+ 375us) server_negotiation.cc:407] Connection header received
0620 18:36:41.838658 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:41.838665 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:41.838747 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:41.838842 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:41.839543 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:41.840244 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:41.841445 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:41.841871 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:41.841964 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:41.845960 (+ 3996us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.37.3:42519: BlockingRecv error: recv got EOF from 127.22.37.3:42519 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Jun 20 18:36:41 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.37.1@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:36:41 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.37.1@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:36:41.952929 31910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:41.944721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:40737 (local address 127.22.37.1:57953)
0620 18:36:41.945290 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:41.945306 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:41.945471 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:41.945878 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:41.945885 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:41.946348 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:41.946493 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:41.946502 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:41.947499 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:41.947505 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:41.948434 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:41.948446 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:41.949166 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:41.949173 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:41.949350 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:41.949982 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:41.950000 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:41.952688 (+ 2688us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.37.62:40737: Server oryx/127.22.37.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":298,"threads_started":1}
W20260620 18:36:41.953351 31905 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:36:41.944857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.1:57953 (local address 127.22.37.62:40737)
0620 18:36:41.945114 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:41.945119 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:41.945408 (+ 289us) server_negotiation.cc:407] Connection header received
0620 18:36:41.945599 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:41.945604 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:41.945691 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:41.945826 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:41.946633 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:41.947357 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:41.948591 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:41.949013 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:41.949148 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:41.952943 (+ 3795us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.37.1:57953: BlockingRecv error: recv got EOF from 127.22.37.1:57953 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Jun 20 18:36:42 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.37.2@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:36:42 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.37.2@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:36:42.406548 31911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:42.398027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:40737 (local address 127.22.37.2:42729)
0620 18:36:42.398577 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:42.398593 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:42.398709 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:42.399038 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:42.399041 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:42.399422 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:42.399562 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:42.399569 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:42.400558 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:42.400562 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:42.401584 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:42.401605 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:42.402425 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:42.402429 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:42.402594 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:42.403421 (+ 827us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:42.403438 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:42.406209 (+ 2771us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.37.62:40737: Server oryx/127.22.37.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":312,"threads_started":1}
W20260620 18:36:42.406592 31905 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:36:42.398118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.2:42729 (local address 127.22.37.62:40737)
0620 18:36:42.398345 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:42.398352 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:42.398674 (+ 322us) server_negotiation.cc:407] Connection header received
0620 18:36:42.398823 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:42.398827 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:42.398901 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:42.399007 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:42.399697 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:42.400422 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:42.401834 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:42.402268 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:42.402416 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:42.406392 (+ 3976us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.37.2:42729: BlockingRecv error: recv got EOF from 127.22.37.2:42729 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Jun 20 18:36:42 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.37.3@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:36:42 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.37.3@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:36:42.855330 31912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:42.846956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:40737 (local address 127.22.37.3:40287)
0620 18:36:42.847487 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:42.847503 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:42.847616 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:42.847931 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:42.847934 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:42.848311 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:42.848469 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:42.848477 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:42.849566 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:42.849571 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:42.850763 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:42.850774 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:42.851605 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:42.851609 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:42.851737 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:42.852280 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:42.852296 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:42.855112 (+ 2816us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.37.62:40737: Server oryx/127.22.37.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":298,"threads_started":1}
W20260620 18:36:42.855670 31905 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:36:42.847163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.3:40287 (local address 127.22.37.62:40737)
0620 18:36:42.847347 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:42.847351 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:42.847615 (+ 264us) server_negotiation.cc:407] Connection header received
0620 18:36:42.847716 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:42.847720 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:42.847801 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:42.847889 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:42.848598 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:42.849357 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:42.850917 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:42.851454 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:42.851570 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:42.855421 (+ 3851us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.37.3:40287: BlockingRecv error: recv got EOF from 127.22.37.3:40287 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Jun 20 18:36:42 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.37.1@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:36:42 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.37.1@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:36:42.962517 31913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:42.953882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:40737 (local address 127.22.37.1:56175)
0620 18:36:42.954465 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:42.954481 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:42.954594 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:42.954880 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:42.954884 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:42.955204 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:42.955372 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:42.955380 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:42.956307 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:42.956310 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:42.957209 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:42.957220 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:42.957866 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:42.957870 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:42.958023 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:42.958768 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:42.958805 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:42.962253 (+ 3448us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.37.62:40737: Server oryx/127.22.37.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":308,"threads_started":1}
W20260620 18:36:42.962849 31905 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:36:42.953943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.1:56175 (local address 127.22.37.62:40737)
0620 18:36:42.954112 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:42.954116 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:42.954552 (+ 436us) server_negotiation.cc:407] Connection header received
0620 18:36:42.954691 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:42.954695 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:42.954751 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:42.954832 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:42.955491 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:42.956185 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:42.957326 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:42.957730 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:42.957863 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:42.962489 (+ 4626us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.37.1:56175: BlockingRecv error: recv got EOF from 127.22.37.1:56175 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Jun 20 18:36:43 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.37.2@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:36:43 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.37.2@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:36:43.417091 31914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:43.407532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:40737 (local address 127.22.37.2:59367)
0620 18:36:43.408092 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:43.408109 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:43.408231 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:43.408609 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:43.408612 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:43.408939 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:43.409099 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:43.409106 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:43.410831 (+ 1725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:43.410834 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:43.411724 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:43.411733 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:43.412857 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:43.412860 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:43.412976 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:43.413652 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:43.413669 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:43.416849 (+ 3180us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.37.62:40737: Server oryx/127.22.37.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":268,"threads_started":1}
W20260620 18:36:43.417448 31905 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:36:43.407637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.2:59367 (local address 127.22.37.62:40737)
0620 18:36:43.407879 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:43.407887 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:43.408193 (+ 306us) server_negotiation.cc:407] Connection header received
0620 18:36:43.408352 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:43.408360 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:43.408461 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:43.408591 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:43.409247 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:43.410604 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:43.411865 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:43.412700 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:43.412871 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:43.417116 (+ 4245us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.37.2:59367: BlockingRecv error: recv got EOF from 127.22.37.2:59367 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Jun 20 18:36:43 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.37.3@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:36:43 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.37.3@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:36:43.864856 31915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:43.856253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:40737 (local address 127.22.37.3:53505)
0620 18:36:43.856765 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:43.856804 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:43.856982 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:43.857335 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:43.857339 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:43.857704 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:43.857945 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:43.857960 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:43.859116 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:43.859123 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:43.860097 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:43.860109 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:43.860827 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:43.860832 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:43.860993 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:43.861696 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:43.861714 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:43.864473 (+ 2759us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.37.62:40737: Server oryx/127.22.37.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":278,"threads_started":1}
W20260620 18:36:43.864903 31905 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:36:43.856333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.3:53505 (local address 127.22.37.62:40737)
0620 18:36:43.856593 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:43.856602 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:43.856897 (+ 295us) server_negotiation.cc:407] Connection header received
0620 18:36:43.857113 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:43.857118 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:43.857198 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:43.857311 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:43.858112 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:43.858973 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:43.860231 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:43.860654 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:43.860788 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:43.864673 (+ 3885us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.37.3:53505: BlockingRecv error: recv got EOF from 127.22.37.3:53505 (error 108)
Metrics: {"server-negotiator.queue_time_us":107}
Jun 20 18:36:43 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.37.1@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:36:43 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.37.1@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:36:43.972270 31916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:43.963591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:40737 (local address 127.22.37.1:39313)
0620 18:36:43.964161 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:43.964181 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:43.964353 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:43.964717 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:43.964721 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:43.965111 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:43.965254 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:43.965262 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:43.966549 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:43.966554 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:43.967863 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:43.967882 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:43.968497 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:43.968501 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:43.968621 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:43.969290 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:43.969308 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:43.972042 (+ 2734us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.37.62:40737: Server oryx/127.22.37.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":277,"threads_started":1}
W20260620 18:36:43.972608 31905 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:36:43.963697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.1:39313 (local address 127.22.37.62:40737)
0620 18:36:43.964001 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:43.964007 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:43.964270 (+ 263us) server_negotiation.cc:407] Connection header received
0620 18:36:43.964504 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:43.964509 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:43.964579 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:43.964676 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:43.965420 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:43.966391 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:43.967993 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:43.968385 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:43.968483 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:43.972218 (+ 3735us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.37.1:39313: BlockingRecv error: recv got EOF from 127.22.37.1:39313 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Jun 20 18:36:44 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.37.2@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:36:44 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.37.2@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:36:44.426779 31917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:44.418133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:40737 (local address 127.22.37.2:52303)
0620 18:36:44.418595 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:44.418613 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:44.418740 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:44.419131 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:44.419134 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:44.419472 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:44.419613 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:44.419620 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:44.420615 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:44.420619 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:44.421754 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:44.421768 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:44.422468 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:44.422473 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:44.422633 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:44.423263 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:44.423283 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:44.426546 (+ 3263us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.37.62:40737: Server oryx/127.22.37.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":245,"threads_started":1}
W20260620 18:36:44.426909 31905 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:36:44.418221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.2:52303 (local address 127.22.37.62:40737)
0620 18:36:44.418478 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:44.418485 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:44.418687 (+ 202us) server_negotiation.cc:407] Connection header received
0620 18:36:44.418926 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:44.418930 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:44.419005 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:44.419087 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:44.419739 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:44.420465 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:44.421896 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:44.422329 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:44.422424 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:44.426713 (+ 4289us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.37.2:52303: BlockingRecv error: recv got EOF from 127.22.37.2:52303 (error 108)
Metrics: {"server-negotiator.queue_time_us":110}
Jun 20 18:36:44 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.37.3@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:36:44 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.37.3@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:36:44.874374 31918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:44.865931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:40737 (local address 127.22.37.3:36981)
0620 18:36:44.866447 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:44.866463 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:44.866587 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:44.866980 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:44.866983 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:44.867327 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:44.867493 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:44.867501 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:44.868431 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:44.868435 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:44.869556 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:44.869602 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:44.870457 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:44.870463 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:44.870656 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:44.871359 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:44.871377 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:44.874146 (+ 2769us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.37.62:40737: Server oryx/127.22.37.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":288,"threads_started":1}
W20260620 18:36:44.874547 31905 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:36:44.866027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.3:36981 (local address 127.22.37.62:40737)
0620 18:36:44.866194 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:44.866199 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:44.866543 (+ 344us) server_negotiation.cc:407] Connection header received
0620 18:36:44.866724 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:44.866728 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:44.866816 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:44.866903 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:44.867616 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:44.868322 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:44.869719 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:44.870297 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:44.870434 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:44.874316 (+ 3882us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.37.3:36981: BlockingRecv error: recv got EOF from 127.22.37.3:36981 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 20 18:36:44 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.37.1@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:36:44 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.37.1@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:36:44.984050 31919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:44.973225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:40737 (local address 127.22.37.1:50473)
0620 18:36:44.973750 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:44.973767 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:44.973913 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:44.974201 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:44.974204 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:44.974613 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:44.974771 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:44.974780 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:44.975791 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:44.975794 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:44.976736 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:44.976747 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:44.977418 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:44.977425 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:44.977621 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:44.980754 (+ 3133us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:44.980771 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:44.983701 (+ 2930us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.37.62:40737: Server oryx/127.22.37.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":294,"threads_started":1}
W20260620 18:36:44.984155 31905 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:36:44.973302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.1:50473 (local address 127.22.37.62:40737)
0620 18:36:44.973556 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:44.973563 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:44.973872 (+ 309us) server_negotiation.cc:407] Connection header received
0620 18:36:44.974012 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:44.974015 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:44.974075 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:44.974155 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:44.974935 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:44.975673 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:44.976849 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:44.977241 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:44.977370 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:44.983915 (+ 6545us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.37.1:50473: BlockingRecv error: recv got EOF from 127.22.37.1:50473 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Jun 20 18:36:45 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.37.2@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:36:45 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.37.2@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:36:45.436486 31920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:45.428034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:40737 (local address 127.22.37.2:55217)
0620 18:36:45.428576 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:45.428594 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:45.428724 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:45.429057 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:45.429061 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:45.429557 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:45.429756 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:45.429773 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:45.431119 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:45.431124 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:45.432134 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:45.432146 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:45.432857 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:45.432863 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:45.432980 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:45.433621 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:45.433658 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:45.436241 (+ 2583us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.37.62:40737: Server oryx/127.22.37.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":296,"threads_started":1}
W20260620 18:36:45.436627 31905 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:36:45.428234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.2:55217 (local address 127.22.37.62:40737)
0620 18:36:45.428464 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:45.428470 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:45.428681 (+ 211us) server_negotiation.cc:407] Connection header received
0620 18:36:45.428833 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:45.428836 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:45.428896 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:45.428981 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:45.429935 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:45.430973 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:45.432252 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:45.432734 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:45.432827 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:45.436425 (+ 3598us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.37.2:55217: BlockingRecv error: recv got EOF from 127.22.37.2:55217 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Jun 20 18:36:45 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.37.3@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:36:45 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.37.3@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:36:45.883497 31921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:45.875404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:40737 (local address 127.22.37.3:44547)
0620 18:36:45.875898 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:45.875917 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:45.876028 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:45.876338 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:45.876341 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:45.876733 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:45.876868 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:45.876876 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:45.878255 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:45.878259 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:45.879199 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:45.879210 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:45.879959 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:45.879964 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:45.880086 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:45.880619 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:45.880636 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:45.883265 (+ 2629us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.37.62:40737: Server oryx/127.22.37.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":272,"threads_started":1}
W20260620 18:36:45.883723 31905 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:36:45.875490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.3:44547 (local address 127.22.37.62:40737)
0620 18:36:45.875666 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:45.875670 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:45.875992 (+ 322us) server_negotiation.cc:407] Connection header received
0620 18:36:45.876126 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:45.876130 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:45.876217 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:45.876298 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:45.876994 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:45.878100 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:45.879415 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:45.879788 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:45.879944 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:45.883441 (+ 3497us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.37.3:44547: BlockingRecv error: recv got EOF from 127.22.37.3:44547 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Jun 20 18:36:45 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.37.1@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:36:45 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.37.1@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:36:45.993240 31922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:45.985010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:40737 (local address 127.22.37.1:46497)
0620 18:36:45.985535 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:45.985550 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:45.985662 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:45.985930 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:45.985933 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:45.986312 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:45.986543 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:45.986552 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:45.987453 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:45.987456 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:45.988710 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:45.988724 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:45.989698 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:45.989703 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:45.989829 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:45.990362 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:45.990377 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:45.993016 (+ 2639us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.37.62:40737: Server oryx/127.22.37.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":284,"threads_started":1}
W20260620 18:36:45.993592 31905 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:36:45.985049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.1:46497 (local address 127.22.37.62:40737)
0620 18:36:45.985216 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:45.985220 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:45.985621 (+ 401us) server_negotiation.cc:407] Connection header received
0620 18:36:45.985752 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:45.985756 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:45.985814 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:45.985890 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:45.986655 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:45.987337 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:45.988863 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:45.989545 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:45.989639 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:45.993307 (+ 3668us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.37.1:46497: BlockingRecv error: recv got EOF from 127.22.37.1:46497 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 20 18:36:46 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.37.2@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:36:46 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.37.2@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:36:46.445819 31923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:46.437530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:40737 (local address 127.22.37.2:41843)
0620 18:36:46.438101 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:46.438118 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:46.438233 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:46.438543 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:46.438546 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:46.438895 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:46.439066 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:46.439074 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:46.440007 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:46.440010 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:46.440872 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:46.440880 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:46.441609 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:46.441613 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:46.441730 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:46.442421 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:46.442438 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:46.445496 (+ 3058us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.37.62:40737: Server oryx/127.22.37.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":340,"threads_started":1}
W20260620 18:36:46.445894 31905 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:36:46.437607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.2:41843 (local address 127.22.37.62:40737)
0620 18:36:46.437789 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:46.437794 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:46.438208 (+ 414us) server_negotiation.cc:407] Connection header received
0620 18:36:46.438337 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:46.438341 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:46.438402 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:46.438488 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:46.439190 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:46.439865 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:46.441025 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:46.441473 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:46.441566 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:46.445682 (+ 4116us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.37.2:41843: BlockingRecv error: recv got EOF from 127.22.37.2:41843 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 20 18:36:46 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.37.3@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:36:46 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.37.3@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:36:46.893253 31924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:46.884620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:40737 (local address 127.22.37.3:48911)
0620 18:36:46.885174 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:46.885197 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:46.885351 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:46.885787 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:46.885791 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:46.886368 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:46.886569 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:46.886578 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:46.887588 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:46.887591 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:46.888447 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:46.888458 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:46.889178 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:46.889183 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:46.889308 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:46.890023 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:46.890041 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:46.893033 (+ 2992us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.37.62:40737: Server oryx/127.22.37.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":425,"threads_started":1}
W20260620 18:36:46.893559 31905 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:36:46.884855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.3:48911 (local address 127.22.37.62:40737)
0620 18:36:46.885040 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:46.885049 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:46.885293 (+ 244us) server_negotiation.cc:407] Connection header received
0620 18:36:46.885538 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:46.885548 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:46.885621 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:46.885743 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:46.886700 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:46.887455 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:46.888577 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:46.889020 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:46.889113 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:46.893277 (+ 4164us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.37.3:48911: BlockingRecv error: recv got EOF from 127.22.37.3:48911 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 20 18:36:46 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.37.1@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:36:46 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.37.1@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:36:47.003268 31925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:46.994255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:40737 (local address 127.22.37.1:53991)
0620 18:36:46.994926 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:46.994951 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:46.995084 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:46.995372 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:46.995376 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:46.995829 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:46.996022 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:46.996032 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:46.997023 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:46.997027 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:46.998067 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:46.998080 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:46.998765 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:46.998770 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:46.998913 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:46.999599 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:46.999647 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:47.002931 (+ 3284us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.37.62:40737: Server oryx/127.22.37.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":299,"threads_started":1}
W20260620 18:36:47.003369 31905 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:36:46.994301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.1:53991 (local address 127.22.37.62:40737)
0620 18:36:46.994468 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:46.994472 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:46.995029 (+ 557us) server_negotiation.cc:407] Connection header received
0620 18:36:46.995186 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:46.995190 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:46.995249 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:46.995325 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:46.996146 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:46.996859 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:46.998193 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:46.998610 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:46.998702 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:47.003148 (+ 4446us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.37.1:53991: BlockingRecv error: recv got EOF from 127.22.37.1:53991 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 20 18:36:47 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.37.2@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:36:47 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.37.2@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:36:47.455660 31926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:47.446876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:40737 (local address 127.22.37.2:50431)
0620 18:36:47.447426 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:47.447442 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:47.447605 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:47.447987 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:47.447991 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:47.448420 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:47.448579 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:47.448588 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:47.449650 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:47.449658 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:47.450582 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:47.450593 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:47.451611 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:47.451616 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:47.451771 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:47.452513 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:47.452538 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:47.455428 (+ 2890us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.37.62:40737: Server oryx/127.22.37.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":249,"threads_started":1}
W20260620 18:36:47.455914 31905 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:36:47.446952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.2:50431 (local address 127.22.37.62:40737)
0620 18:36:47.447225 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:47.447234 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:47.447538 (+ 304us) server_negotiation.cc:407] Connection header received
0620 18:36:47.447736 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:47.447748 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:47.447808 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:47.447909 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:47.448732 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:47.449510 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:47.450737 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:47.451456 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:47.451555 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:47.455647 (+ 4092us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.37.2:50431: BlockingRecv error: recv got EOF from 127.22.37.2:50431 (error 108)
Metrics: {"server-negotiator.queue_time_us":127}
Jun 20 18:36:47 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.37.3@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:36:47 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.37.3@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:36:47.902611 31927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:47.894197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:40737 (local address 127.22.37.3:37613)
0620 18:36:47.894800 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:47.894817 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:47.894937 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:47.895231 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:47.895234 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:47.895622 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:47.895776 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:47.895784 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:47.896774 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:47.896778 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:47.897723 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:47.897734 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:47.898420 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:47.898425 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:47.898636 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:47.899227 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:47.899244 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:47.902253 (+ 3009us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.37.62:40737: Server oryx/127.22.37.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":278,"threads_started":1}
W20260620 18:36:47.902719 31905 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:36:47.894263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.3:37613 (local address 127.22.37.62:40737)
0620 18:36:47.894477 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:47.894481 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:47.894896 (+ 415us) server_negotiation.cc:407] Connection header received
0620 18:36:47.895041 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:47.895045 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:47.895105 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:47.895189 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:47.895911 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:47.896651 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:47.897844 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:47.898262 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:47.898365 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:47.902434 (+ 4069us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.37.3:37613: BlockingRecv error: recv got EOF from 127.22.37.3:37613 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Jun 20 18:36:48 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.37.1@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:36:48 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.37.1@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:36:48.012270 31928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:48.004215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:40737 (local address 127.22.37.1:33721)
0620 18:36:48.004755 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:48.004771 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:48.004898 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:48.005186 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:48.005189 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:48.005725 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:48.005893 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:48.005903 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:48.006906 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:48.006909 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:48.007797 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:48.007808 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:48.008499 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:48.008503 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:48.008616 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:48.009202 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:48.009219 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:48.012045 (+ 2826us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.37.62:40737: Server oryx/127.22.37.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":285,"threads_started":1}
W20260620 18:36:48.012490 31905 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:36:48.004389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.1:33721 (local address 127.22.37.62:40737)
0620 18:36:48.004557 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:48.004561 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:48.004861 (+ 300us) server_negotiation.cc:407] Connection header received
0620 18:36:48.005001 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:48.005005 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:48.005066 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:48.005148 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:48.006012 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:48.006738 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:48.007918 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:48.008353 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:48.008445 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:48.012258 (+ 3813us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.37.1:33721: BlockingRecv error: recv got EOF from 127.22.37.1:33721 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 20 18:36:48 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.37.2@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:36:48 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.37.2@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:36:48.465212 31929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:48.456664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:40737 (local address 127.22.37.2:56363)
0620 18:36:48.457238 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:48.457254 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:48.457445 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:48.457879 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:48.457886 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:48.458294 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:48.458433 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:48.458441 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:48.459681 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:48.459686 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:48.460572 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:48.460582 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:48.461304 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:48.461309 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:48.461499 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:48.462123 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:48.462140 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:48.464874 (+ 2734us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.37.62:40737: Server oryx/127.22.37.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":314,"threads_started":1}
W20260620 18:36:48.465394 31905 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:36:48.456825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.2:56363 (local address 127.22.37.62:40737)
0620 18:36:48.457068 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:48.457074 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:48.457350 (+ 276us) server_negotiation.cc:407] Connection header received
0620 18:36:48.457647 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:48.457653 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:48.457735 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:48.457841 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:48.458572 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:48.459539 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:48.460708 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:48.461131 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:48.461271 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:48.465120 (+ 3849us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.37.2:56363: BlockingRecv error: recv got EOF from 127.22.37.2:56363 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Jun 20 18:36:48 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.37.3@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:36:48 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.37.3@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:36:48.912725 31930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:48.903643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:40737 (local address 127.22.37.3:37803)
0620 18:36:48.904241 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:48.904259 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:48.904431 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:48.904814 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:48.904821 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:48.905273 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:48.905535 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:48.905546 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:48.906615 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:48.906622 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:48.907597 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:48.907609 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:48.908416 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:48.908423 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:48.908620 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:48.909277 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:48.909305 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:48.912415 (+ 3110us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.37.62:40737: Server oryx/127.22.37.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":288,"threads_started":1}
W20260620 18:36:48.912906 31905 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:36:48.903822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.3:37803 (local address 127.22.37.62:40737)
0620 18:36:48.904070 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:48.904077 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:48.904353 (+ 276us) server_negotiation.cc:407] Connection header received
0620 18:36:48.904591 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:48.904596 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:48.904660 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:48.904770 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:48.905688 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:48.906448 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:48.907765 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:48.908209 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:48.908377 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:48.912657 (+ 4280us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.37.3:37803: BlockingRecv error: recv got EOF from 127.22.37.3:37803 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Jun 20 18:36:49 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.37.1@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:36:49 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.37.1@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:36:49.021520 31931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:49.013298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:40737 (local address 127.22.37.1:38131)
0620 18:36:49.013869 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:49.013885 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:49.014034 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:49.014374 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:49.014378 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:49.014765 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:49.014936 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:49.014944 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:49.015987 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:49.015991 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:49.016948 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:49.016960 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:49.017808 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:49.017813 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:49.017948 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:49.018533 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:49.018550 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:49.021201 (+ 2651us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.37.62:40737: Server oryx/127.22.37.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":347,"threads_started":1}
W20260620 18:36:49.021783 31905 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:36:49.013410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.1:38131 (local address 127.22.37.62:40737)
0620 18:36:49.013673 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:49.013681 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:49.013989 (+ 308us) server_negotiation.cc:407] Connection header received
0620 18:36:49.014138 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:49.014142 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:49.014209 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:49.014314 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:49.015072 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:49.015833 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:49.017095 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:49.017636 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:49.017751 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:49.021485 (+ 3734us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.37.1:38131: BlockingRecv error: recv got EOF from 127.22.37.1:38131 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Jun 20 18:36:49 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.37.2@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:36:49 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.37.2@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:36:49.474627 31932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:49.466205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:40737 (local address 127.22.37.2:53061)
0620 18:36:49.466816 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:49.466833 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:49.466943 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:49.467234 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:49.467237 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:49.467599 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:49.467751 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:49.467758 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:49.468773 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:49.468778 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:49.469941 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:49.469955 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:49.470689 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:49.470693 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:49.470815 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:49.471374 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:49.471390 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:49.474369 (+ 2979us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.37.62:40737: Server oryx/127.22.37.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":378,"threads_started":1}
W20260620 18:36:49.474926 31905 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:36:49.466290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.2:53061 (local address 127.22.37.62:40737)
0620 18:36:49.466496 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:49.466504 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:49.466929 (+ 425us) server_negotiation.cc:407] Connection header received
0620 18:36:49.467040 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:49.467044 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:49.467104 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:49.467185 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:49.467874 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:49.468626 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:49.470111 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:49.470540 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:49.470635 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:49.474662 (+ 4027us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.37.2:53061: BlockingRecv error: recv got EOF from 127.22.37.2:53061 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 20 18:36:49 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.37.3@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:36:49 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.37.3@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:36:49.922073 31933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:49.913826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:40737 (local address 127.22.37.3:46689)
0620 18:36:49.914382 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:49.914398 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:49.914517 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:49.914803 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:49.914806 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:49.915204 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:49.915343 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:49.915351 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:49.916313 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:49.916316 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:49.917271 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:49.917281 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:49.918011 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:49.918015 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:49.918135 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:49.918708 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:49.918725 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:49.921642 (+ 2917us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.37.62:40737: Server oryx/127.22.37.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":305,"threads_started":1}
W20260620 18:36:49.922219 31905 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:36:49.913936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.3:46689 (local address 127.22.37.62:40737)
0620 18:36:49.914135 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:49.914140 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:49.914470 (+ 330us) server_negotiation.cc:407] Connection header received
0620 18:36:49.914617 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:49.914621 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:49.914679 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:49.914763 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:49.915476 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:49.916202 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:49.917464 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:49.917863 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:49.917960 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:49.921960 (+ 4000us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.37.3:46689: BlockingRecv error: recv got EOF from 127.22.37.3:46689 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 20 18:36:50 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.37.1@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:36:50 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.37.1@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:36:50.034865 31934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:50.022445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:40737 (local address 127.22.37.1:50653)
0620 18:36:50.023073 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:50.023089 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:50.023195 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:50.023503 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:50.023506 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:50.023846 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:50.023991 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:50.023999 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:50.024961 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:50.024964 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:50.025888 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:50.025898 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:50.026578 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:50.026582 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:50.026701 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:50.031439 (+ 4738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:50.031461 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:50.034625 (+ 3164us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.37.62:40737: Server oryx/127.22.37.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":404,"threads_started":1}
W20260620 18:36:50.035084 31905 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:36:50.022607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.1:50653 (local address 127.22.37.62:40737)
0620 18:36:50.022771 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:50.022775 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:50.023170 (+ 395us) server_negotiation.cc:407] Connection header received
0620 18:36:50.023292 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:50.023296 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:50.023355 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:50.023446 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:50.024112 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:50.024844 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:50.026012 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:50.026450 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:50.026542 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:50.034832 (+ 8290us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.37.1:50653: BlockingRecv error: recv got EOF from 127.22.37.1:50653 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 20 18:36:50 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.37.2@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:36:50 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.37.2@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:36:50.483906 31935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:50.475547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:40737 (local address 127.22.37.2:44077)
0620 18:36:50.476062 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:50.476079 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:50.476190 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:50.476516 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:50.476519 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:50.476867 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:50.477008 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:50.477016 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:50.478145 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:50.478149 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:50.479220 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:50.479237 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:50.479917 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:50.479921 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:50.480040 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:50.480690 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:50.480707 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:50.483660 (+ 2953us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.37.62:40737: Server oryx/127.22.37.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":303,"threads_started":1}
W20260620 18:36:50.484120 31905 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:36:50.475663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.2:44077 (local address 127.22.37.62:40737)
0620 18:36:50.475879 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:50.475883 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:50.476182 (+ 299us) server_negotiation.cc:407] Connection header received
0620 18:36:50.476306 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:50.476310 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:50.476377 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:50.476473 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:50.477156 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:50.477981 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:50.479369 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:50.479777 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:50.479892 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:50.483858 (+ 3966us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.37.2:44077: BlockingRecv error: recv got EOF from 127.22.37.2:44077 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
Jun 20 18:36:50 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.37.3@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:36:50 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.37.3@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:36:50.930923 31936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:50.923243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:40737 (local address 127.22.37.3:45503)
0620 18:36:50.923786 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:50.923801 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:50.923904 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:50.924174 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:50.924177 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:50.924544 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:50.924694 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:50.924702 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:50.925712 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:50.925717 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:50.926619 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:50.926630 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:50.927296 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:50.927300 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:50.927414 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:50.928077 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:50.928094 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:50.930699 (+ 2605us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.37.62:40737: Server oryx/127.22.37.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":325,"threads_started":1}
W20260620 18:36:50.931134 31905 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:36:50.923428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.3:45503 (local address 127.22.37.62:40737)
0620 18:36:50.923629 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:50.923633 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:50.923869 (+ 236us) server_negotiation.cc:407] Connection header received
0620 18:36:50.923988 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:50.923991 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:50.924058 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:50.924132 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:50.924835 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:50.925585 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:50.926753 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:50.927175 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:50.927268 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:50.930878 (+ 3610us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.37.3:45503: BlockingRecv error: recv got EOF from 127.22.37.3:45503 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
Jun 20 18:36:51 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.37.1@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:36:51 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.37.1@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:36:51.044255 31937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:51.035747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:40737 (local address 127.22.37.1:39109)
0620 18:36:51.036327 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:51.036353 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:51.036472 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:51.036752 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:51.036756 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:51.037127 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:51.037296 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:51.037303 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:51.038326 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:51.038334 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:51.039789 (+ 1455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:51.039801 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:51.040490 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:51.040494 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:51.040598 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:51.041155 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:51.041172 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:51.044029 (+ 2857us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.37.62:40737: Server oryx/127.22.37.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":332,"threads_started":1}
W20260620 18:36:51.044476 31905 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:36:51.035864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.1:39109 (local address 127.22.37.62:40737)
0620 18:36:51.036087 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:51.036092 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:51.036433 (+ 341us) server_negotiation.cc:407] Connection header received
0620 18:36:51.036565 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:51.036568 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:51.036628 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:51.036713 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:51.037458 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:51.038203 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:51.039921 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:51.040343 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:51.040439 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:51.044245 (+ 3806us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.37.1:39109: BlockingRecv error: recv got EOF from 127.22.37.1:39109 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Jun 20 18:36:51 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.37.2@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:36:51 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.37.2@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:36:51.493728 31938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:51.484939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:40737 (local address 127.22.37.2:50603)
0620 18:36:51.485495 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:51.485512 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:51.485629 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:51.485978 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:51.485982 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:51.486381 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:51.486560 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:51.486569 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:51.487562 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:51.487566 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:51.488625 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:51.488639 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:51.489634 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:51.489642 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:51.489869 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:51.490464 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:51.490482 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:51.493371 (+ 2889us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.37.62:40737: Server oryx/127.22.37.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":301,"threads_started":1}
W20260620 18:36:51.493844 31905 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:36:51.485017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.2:50603 (local address 127.22.37.62:40737)
0620 18:36:51.485242 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:51.485250 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:51.485591 (+ 341us) server_negotiation.cc:407] Connection header received
0620 18:36:51.485731 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:51.485735 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:51.485798 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:51.485886 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:51.486691 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:51.487452 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:51.488800 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:51.489440 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:51.489607 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:51.493611 (+ 4004us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.37.2:50603: BlockingRecv error: recv got EOF from 127.22.37.2:50603 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 20 18:36:51 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.37.3@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:36:51 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.37.3@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:36:51.940332 31939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:51.931871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:40737 (local address 127.22.37.3:36275)
0620 18:36:51.932418 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:51.932434 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:51.932550 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:51.933031 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:51.933035 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:51.933490 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:51.933709 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:51.933728 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:51.934768 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:51.934774 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:51.935685 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:51.935696 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:51.936438 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:51.936444 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:51.936613 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:51.937221 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:51.937239 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:51.939989 (+ 2750us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.37.62:40737: Server oryx/127.22.37.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":265,"threads_started":1}
W20260620 18:36:51.940379 31905 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:36:51.931925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.3:36275 (local address 127.22.37.62:40737)
0620 18:36:51.932116 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:51.932126 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:51.932507 (+ 381us) server_negotiation.cc:407] Connection header received
0620 18:36:51.932715 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:51.932719 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:51.932776 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:51.932853 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:51.933877 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:51.934624 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:51.935826 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:51.936252 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:51.936393 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:51.940166 (+ 3773us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.37.3:36275: BlockingRecv error: recv got EOF from 127.22.37.3:36275 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Jun 20 18:36:52 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.37.1@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:36:52 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.37.1@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:36:52.053710 31940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:52.045226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:40737 (local address 127.22.37.1:51399)
0620 18:36:52.045837 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:52.045854 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:52.045965 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:52.046279 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:52.046283 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:52.046704 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:52.046853 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:52.046861 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:52.047877 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:52.047880 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:52.048753 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:52.048763 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:52.049663 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:52.049669 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:52.049857 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:52.050602 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:52.050620 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:52.053449 (+ 2829us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.37.62:40737: Server oryx/127.22.37.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":374,"threads_started":1}
W20260620 18:36:52.053939 31905 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:36:52.045263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.1:51399 (local address 127.22.37.62:40737)
0620 18:36:52.045521 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:52.045525 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:52.045936 (+ 411us) server_negotiation.cc:407] Connection header received
0620 18:36:52.046068 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:52.046071 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:52.046145 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:52.046238 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:52.046983 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:52.047764 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:52.048882 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:52.049438 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:52.049631 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:52.053671 (+ 4040us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.37.1:51399: BlockingRecv error: recv got EOF from 127.22.37.1:51399 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 20 18:36:52 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.37.2@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:36:52 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.37.2@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:36:52.502478 31941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:52.494661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:40737 (local address 127.22.37.2:40161)
0620 18:36:52.495176 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:52.495193 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:52.495315 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:52.495606 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:52.495609 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:52.495964 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:52.496109 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:52.496117 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:52.497095 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:52.497099 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:52.498085 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:52.498097 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:52.498721 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:52.498725 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:52.498846 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:52.499512 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:52.499529 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:52.502244 (+ 2715us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.37.62:40737: Server oryx/127.22.37.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":299,"threads_started":1}
W20260620 18:36:52.502676 31905 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:36:52.494686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.2:40161 (local address 127.22.37.62:40737)
0620 18:36:52.494955 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:52.494959 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:52.495275 (+ 316us) server_negotiation.cc:407] Connection header received
0620 18:36:52.495415 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:52.495418 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:52.495480 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:52.495557 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:52.496236 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:52.496940 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:52.498220 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:52.498598 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:52.498691 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:52.502416 (+ 3725us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.37.2:40161: BlockingRecv error: recv got EOF from 127.22.37.2:40161 (error 108)
Metrics: {"server-negotiator.queue_time_us":161}
Jun 20 18:36:52 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.37.3@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:36:52 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.37.3@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:36:52.949990 31942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:52.941444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:40737 (local address 127.22.37.3:38429)
0620 18:36:52.941962 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:52.941978 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:52.942098 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:52.942544 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:52.942586 (+ 42us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:52.943134 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:52.943350 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:52.943360 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:52.944610 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:52.944614 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:52.945603 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:52.945615 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:52.946244 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:52.946248 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:52.946365 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:52.947011 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:52.947029 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:52.949755 (+ 2726us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.37.62:40737: Server oryx/127.22.37.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":288,"threads_started":1}
W20260620 18:36:52.950194 31905 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:36:52.941526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.3:38429 (local address 127.22.37.62:40737)
0620 18:36:52.941772 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:52.941779 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:52.942091 (+ 312us) server_negotiation.cc:407] Connection header received
0620 18:36:52.942204 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:52.942209 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:52.942308 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:52.942416 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:52.943516 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:52.944477 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:52.945724 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:52.946131 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:52.946221 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:52.949949 (+ 3728us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.37.3:38429: BlockingRecv error: recv got EOF from 127.22.37.3:38429 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Jun 20 18:36:53 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.37.1@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:36:53 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.37.1@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:36:53.063616 31943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:53.054651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:40737 (local address 127.22.37.1:58095)
0620 18:36:53.055161 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:53.055176 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:53.055326 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:53.055649 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:53.055654 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:53.056071 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:53.056226 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:53.056235 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:53.057224 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:53.057230 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:53.058339 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:53.058351 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:53.059037 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:53.059043 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:53.059246 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:53.059866 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:53.059885 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:53.063223 (+ 3338us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.37.62:40737: Server oryx/127.22.37.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":295,"threads_started":1}
W20260620 18:36:53.063727 31905 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:36:53.054724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.1:58095 (local address 127.22.37.62:40737)
0620 18:36:53.054969 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:53.054976 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:53.055259 (+ 283us) server_negotiation.cc:407] Connection header received
0620 18:36:53.055454 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:53.055465 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:53.055522 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:53.055620 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:53.056366 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:53.057080 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:53.058477 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:53.058898 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:53.059027 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:53.063445 (+ 4418us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.37.1:58095: BlockingRecv error: recv got EOF from 127.22.37.1:58095 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Jun 20 18:36:53 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.37.2@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:36:53 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.37.2@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:36:53.511685 31944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:53.503481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:40737 (local address 127.22.37.2:42719)
0620 18:36:53.504036 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:53.504056 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:53.504200 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:53.504602 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:53.504606 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:53.505041 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:53.505222 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:53.505230 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:53.506360 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:53.506364 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:53.507254 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:53.507265 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:53.508003 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:53.508007 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:53.508129 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:53.508716 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:53.508732 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:53.511465 (+ 2733us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.37.62:40737: Server oryx/127.22.37.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":470,"threads_started":1}
W20260620 18:36:53.511956 31905 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:36:53.503697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.2:42719 (local address 127.22.37.62:40737)
0620 18:36:53.503915 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:53.503921 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:53.504128 (+ 207us) server_negotiation.cc:407] Connection header received
0620 18:36:53.504328 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:53.504334 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:53.504436 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:53.504525 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:53.505347 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:53.506244 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:53.507399 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:53.507836 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:53.507934 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:53.511681 (+ 3747us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.37.2:42719: BlockingRecv error: recv got EOF from 127.22.37.2:42719 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Jun 20 18:36:53 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.37.3@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:36:53 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.37.3@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:36:53.959403 31945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:53.950981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:40737 (local address 127.22.37.3:49267)
0620 18:36:53.951527 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:53.951543 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:53.951660 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:53.951945 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:53.951948 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:53.952328 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:53.952465 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:53.952474 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:53.953734 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:53.953738 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:53.954667 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:53.954678 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:53.955460 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:53.955465 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:53.955602 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:53.956189 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:53.956206 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:53.959057 (+ 2851us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.37.62:40737: Server oryx/127.22.37.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":305,"threads_started":1}
W20260620 18:36:53.959618 31905 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:36:53.951039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.3:49267 (local address 127.22.37.62:40737)
0620 18:36:53.951201 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:53.951205 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:53.951624 (+ 419us) server_negotiation.cc:407] Connection header received
0620 18:36:53.951760 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:53.951764 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:53.951821 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:53.951903 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:53.952704 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:53.953591 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:53.954829 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:53.955303 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:53.955422 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:53.959352 (+ 3930us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.37.3:49267: BlockingRecv error: recv got EOF from 127.22.37.3:49267 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 20 18:36:54 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.37.1@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:36:54 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.37.1@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:36:54.073978 31946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:54.064720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:40737 (local address 127.22.37.1:53811)
0620 18:36:54.065222 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:54.065238 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:54.065410 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:54.065754 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:54.065758 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:54.066165 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:54.066329 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:54.066337 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:54.067408 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:54.067414 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:54.069036 (+ 1622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:54.069048 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:54.069895 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:54.069900 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:54.070033 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:54.070721 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:54.070740 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:54.073754 (+ 3014us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.37.62:40737: Server oryx/127.22.37.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":272,"threads_started":1}
W20260620 18:36:54.074347 31905 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:36:54.064796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.1:53811 (local address 127.22.37.62:40737)
0620 18:36:54.065113 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:54.065119 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:54.065321 (+ 202us) server_negotiation.cc:407] Connection header received
0620 18:36:54.065541 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:54.065548 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:54.065610 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:54.065723 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:54.066470 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:54.067197 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:54.069161 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:54.069748 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:54.069850 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:54.073956 (+ 4106us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.37.1:53811: BlockingRecv error: recv got EOF from 127.22.37.1:53811 (error 108)
Metrics: {"server-negotiator.queue_time_us":158}
Jun 20 18:36:54 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.37.2@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:36:54 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.37.2@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:36:54.521296 31947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:54.512753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:40737 (local address 127.22.37.2:57051)
0620 18:36:54.513304 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:54.513322 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:54.513476 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:54.513781 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:54.513784 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:54.514173 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:54.514355 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:54.514364 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:54.515309 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:54.515313 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:54.516248 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:54.516260 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:54.516908 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:54.516912 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:54.517035 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:54.517733 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:54.517750 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:54.520946 (+ 3196us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.37.62:40737: Server oryx/127.22.37.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":317,"threads_started":1}
W20260620 18:36:54.521335 31905 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:36:54.512837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.2:57051 (local address 127.22.37.62:40737)
0620 18:36:54.513025 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:54.513029 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:54.513406 (+ 377us) server_negotiation.cc:407] Connection header received
0620 18:36:54.513578 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:54.513581 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:54.513638 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:54.513718 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:54.514504 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:54.515165 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:54.516380 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:54.516761 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:54.516848 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:54.521125 (+ 4277us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.37.2:57051: BlockingRecv error: recv got EOF from 127.22.37.2:57051 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Jun 20 18:36:54 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.37.3@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:36:54 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.37.3@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:36:54.969425 31948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:54.960708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:40737 (local address 127.22.37.3:41775)
0620 18:36:54.961219 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:54.961235 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:54.961417 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:54.961762 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:54.961767 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:54.962132 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:54.962308 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:54.962316 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:54.963462 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:54.963467 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:54.964510 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:54.964521 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:54.965241 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:54.965247 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:54.965451 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:54.966063 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:54.966084 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:54.969031 (+ 2947us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.37.62:40737: Server oryx/127.22.37.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":294,"threads_started":1}
W20260620 18:36:54.969576 31905 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:36:54.960782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.3:41775 (local address 127.22.37.62:40737)
0620 18:36:54.961012 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:54.961021 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:54.961330 (+ 309us) server_negotiation.cc:407] Connection header received
0620 18:36:54.961559 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:54.961568 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:54.961628 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:54.961731 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:54.962454 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:54.963300 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:54.964644 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:54.965098 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:54.965231 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:54.969246 (+ 4015us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.37.3:41775: BlockingRecv error: recv got EOF from 127.22.37.3:41775 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Jun 20 18:36:55 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.37.1@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:36:55 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.37.1@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:36:55.087744 31949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:55.075080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:40737 (local address 127.22.37.1:34931)
0620 18:36:55.075570 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:55.075585 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:55.075733 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:55.076064 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:55.076069 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:55.076496 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:55.076691 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:55.076700 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:55.077761 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:55.077767 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:55.078682 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:55.078693 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:55.079519 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:55.079525 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:55.079702 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:55.084482 (+ 4780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:55.084509 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:55.087382 (+ 2873us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.37.62:40737: Server oryx/127.22.37.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":257,"threads_started":1}
W20260620 18:36:55.087816 31905 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:36:55.075147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.1:34931 (local address 127.22.37.62:40737)
0620 18:36:55.075412 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:55.075418 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:55.075664 (+ 246us) server_negotiation.cc:407] Connection header received
0620 18:36:55.075857 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:55.075863 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:55.075931 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:55.076033 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:55.076832 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:55.077629 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:55.078826 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:55.079351 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:55.079512 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:55.087596 (+ 8084us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.37.1:34931: BlockingRecv error: recv got EOF from 127.22.37.1:34931 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 20 18:36:55 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.37.2@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:36:55 dist-test-slave-5lgp krb5kdc[30904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.37.2@KRBTEST.COM for oryx/127.22.37.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:36:55.531275 31950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:55.522638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:40737 (local address 127.22.37.2:57967)
0620 18:36:55.523239 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:55.523256 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:55.523374 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:55.523681 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:55.523684 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:55.524105 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:55.524302 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:55.524310 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:55.525553 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:55.525558 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:55.526780 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:55.526793 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:55.527656 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:55.527660 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:55.527775 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:55.528433 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:55.528458 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:55.531029 (+ 2571us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.37.62:40737: Server oryx/127.22.37.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":351,"threads_started":1}
W20260620 18:36:55.531454 31905 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:36:55.522805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.2:57967 (local address 127.22.37.62:40737)
0620 18:36:55.522979 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:55.522983 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:55.523334 (+ 351us) server_negotiation.cc:407] Connection header received
0620 18:36:55.523481 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:55.523485 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:55.523548 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:55.523638 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:55.524453 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:55.525331 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:55.526914 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:55.527518 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:55.527614 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:55.531226 (+ 3612us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.37.2:57967: BlockingRecv error: recv got EOF from 127.22.37.2:57967 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 18:36:55.804817 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 31491
I20260620 18:36:55.818092 31952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:55.806036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53108 (local address 127.22.37.1:45785)
0620 18:36:55.806697 (+ 661us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:55.806706 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:55.806735 (+ 29us) server_negotiation.cc:407] Connection header received
0620 18:36:55.806799 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:55.806806 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:55.807039 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:55.807210 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:55.808690 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:55.809840 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:55.811305 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:55.811946 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:55.812044 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:55.814627 (+ 2583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:55.814640 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:55.814655 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:55.814692 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:55.817287 (+ 2595us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.22.37.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0620 18:36:55.817838 (+ 551us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53108: Request ticket server kudu/127.22.37.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":357,"threads_started":1}
W20260620 18:36:55.818226 31952 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53108: Request ticket server kudu/127.22.37.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260620 18:36:55.818233 22676 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 31491: Not authorized: Client connection negotiation failed: client connection to 127.22.37.1:45785: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.22.37.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260620 18:36:55.818535 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 31491
I20260620 18:36:55.835680 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 31642
I20260620 18:36:55.848340 31953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:55.836726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43576 (local address 127.22.37.2:34745)
0620 18:36:55.837292 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:55.837299 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:55.837329 (+ 30us) server_negotiation.cc:407] Connection header received
0620 18:36:55.837448 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:55.837455 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:55.837680 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:55.837832 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:55.839100 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:55.840178 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:55.841671 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:55.842254 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:55.842358 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:55.844998 (+ 2640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:55.845008 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:55.845023 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:55.845071 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:55.847807 (+ 2736us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.22.37.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0620 18:36:55.848080 (+ 273us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43576: Request ticket server kudu/127.22.37.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":281,"threads_started":1}
W20260620 18:36:55.848534 22676 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 31642: Not authorized: Client connection negotiation failed: client connection to 127.22.37.2:34745: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.22.37.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260620 18:36:55.848726 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 31642
I20260620 18:36:55.865870 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 31774
I20260620 18:36:55.878085 31954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:55.866838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58946 (local address 127.22.37.3:42181)
0620 18:36:55.867475 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:55.867484 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:55.867513 (+ 29us) server_negotiation.cc:407] Connection header received
0620 18:36:55.867577 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:55.867583 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:55.867839 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:55.868029 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:55.869401 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:55.870605 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:55.871852 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:55.872410 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:55.872519 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:55.875123 (+ 2604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:55.875132 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:55.875147 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:55.875182 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:55.877610 (+ 2428us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.22.37.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0620 18:36:55.877852 (+ 242us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58946: Request ticket server kudu/127.22.37.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":321,"threads_started":1}
W20260620 18:36:55.878201 31954 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58946: Request ticket server kudu/127.22.37.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260620 18:36:55.878244 22676 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 31774: Not authorized: Client connection negotiation failed: client connection to 127.22.37.3:42181: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.22.37.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260620 18:36:55.878413 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 31774
I20260620 18:36:55.895370 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 31422
I20260620 18:36:55.896656 31484 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:36:56.013085 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 31422
2026-06-20T18:36:56Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21066 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-20T18:36:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:36:56Z Disabled control of system clock
I20260620 18:36:56.090576 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980527431848-22676-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980527431848-22676-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980527431848-22676-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.62:36583
--webserver_interface=127.22.37.62
--webserver_port=0
--builtin_ntp_servers=127.22.37.20:32873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.37.62:36583
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 18:36:56.415144 31959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:36:56.415521 31959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:36:56.415663 31959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:36:56.423014 31959 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:36:56.423156 31959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:36:56.423235 31959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:36:56.423346 31959 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:36:56.423425 31959 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:36:56.432565 31959 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:32873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980527431848-22676-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980527431848-22676-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.37.62:36583
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.62:36583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.37.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980527431848-22676-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:36:56.433928 31959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:36:56.435678 31959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:36:56.447551 31964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:56.447645 31965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:56.447616 31967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:56.448102 31959 server_base.cc:1034] running on GCE node
I20260620 18:36:56.449070 31959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:36:56.450642 31959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:36:56.451917 31959 hybrid_clock.cc:648] HybridClock initialized: now 1781980616451891 us; error 42 us; skew 500 ppm
I20260620 18:36:56.452282 31959 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:36:56.454725 31959 webserver.cc:460] Webserver started at http://127.22.37.62:37659/ using document root <none> and password file <none>
I20260620 18:36:56.455462 31959 fs_manager.cc:359] Metadata directory not provided
I20260620 18:36:56.455601 31959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:36:56.455960 31959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:36:56.459034 31959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980527431848-22676-0/minicluster-data/master-0/data/instance:
uuid: "d9419dcae5c6468b8ddec76c29872c65"
format_stamp: "Formatted at 2026-06-20 18:36:56 on dist-test-slave-5lgp"
server_key: "803844279bd061a737e940cb3c4347b3"
server_key_iv: "9f1b4fc003b16c24a487a0210ff94829"
server_key_version: "encryptionkey@0"
I20260620 18:36:56.459954 31959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980527431848-22676-0/minicluster-data/master-0/wal/instance:
uuid: "d9419dcae5c6468b8ddec76c29872c65"
format_stamp: "Formatted at 2026-06-20 18:36:56 on dist-test-slave-5lgp"
server_key: "803844279bd061a737e940cb3c4347b3"
server_key_iv: "9f1b4fc003b16c24a487a0210ff94829"
server_key_version: "encryptionkey@0"
I20260620 18:36:56.465284 31959 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 18:36:56.468792 31973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:56.470330 31959 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 18:36:56.470515 31959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980527431848-22676-0/minicluster-data/master-0/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980527431848-22676-0/minicluster-data/master-0/wal
uuid: "d9419dcae5c6468b8ddec76c29872c65"
format_stamp: "Formatted at 2026-06-20 18:36:56 on dist-test-slave-5lgp"
server_key: "803844279bd061a737e940cb3c4347b3"
server_key_iv: "9f1b4fc003b16c24a487a0210ff94829"
server_key_version: "encryptionkey@0"
I20260620 18:36:56.470741 31959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980527431848-22676-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980527431848-22676-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980527431848-22676-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:36:56.489738 31959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:36:56.490703 31959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:36:56.491008 31959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:36:56.509353 31959 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.62:36583
I20260620 18:36:56.509433 32024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.62:36583 every 8 connection(s)
I20260620 18:36:56.510815 31959 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
I20260620 18:36:56.514559 32025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:36:56.521183 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 31959
I20260620 18:36:56.521454 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980527431848-22676-0/minicluster-data/master-0/wal/instance
I20260620 18:36:56.522122 22676 external_mini_cluster.cc:1428] Setting key aa126e0db1fa4b8d1dc36ae116696d99
I20260620 18:36:56.524684 32025 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9419dcae5c6468b8ddec76c29872c65: Bootstrap starting.
I20260620 18:36:56.528056 32025 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d9419dcae5c6468b8ddec76c29872c65: Neither blocks nor log segments found. Creating new log.
I20260620 18:36:56.529204 32025 log.cc:824] T 00000000000000000000000000000000 P d9419dcae5c6468b8ddec76c29872c65: Log is configured to *not* fsync() on all Append() calls
I20260620 18:36:56.532368 32025 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9419dcae5c6468b8ddec76c29872c65: No bootstrap required, opened a new log
I20260620 18:36:56.536238 32028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:56.524168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49950 (local address 127.22.37.62:36583)
0620 18:36:56.525199 (+ 1031us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:56.525210 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:56.525249 (+ 39us) server_negotiation.cc:407] Connection header received
0620 18:36:56.526983 (+ 1734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:56.527004 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:56.527462 (+ 458us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:56.527774 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:56.528511 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:56.530078 (+ 1567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:56.531847 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:56.532543 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:56.532667 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:56.533969 (+ 1302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:56.533976 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:56.533989 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:36:56.534536 (+ 547us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:56.534616 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:36:56.534634 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:56.534723 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:56.534968 (+ 245us) server_negotiation.cc:299] Negotiation successful
0620 18:36:56.535304 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":358,"threads_started":1}
I20260620 18:36:56.540625 32025 raft_consensus.cc:348] T 00000000000000000000000000000000 P d9419dcae5c6468b8ddec76c29872c65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9419dcae5c6468b8ddec76c29872c65" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 36583 } }
I20260620 18:36:56.540977 32025 raft_consensus.cc:374] T 00000000000000000000000000000000 P d9419dcae5c6468b8ddec76c29872c65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:36:56.541095 32025 raft_consensus.cc:729] T 00000000000000000000000000000000 P d9419dcae5c6468b8ddec76c29872c65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9419dcae5c6468b8ddec76c29872c65, State: Initialized, Role: FOLLOWER
I20260620 18:36:56.541687 32025 consensus_queue.cc:260] T 00000000000000000000000000000000 P d9419dcae5c6468b8ddec76c29872c65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9419dcae5c6468b8ddec76c29872c65" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 36583 } }
I20260620 18:36:56.541903 32025 raft_consensus.cc:388] T 00000000000000000000000000000000 P d9419dcae5c6468b8ddec76c29872c65 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:36:56.542040 32025 raft_consensus.cc:482] T 00000000000000000000000000000000 P d9419dcae5c6468b8ddec76c29872c65 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:36:56.542212 32025 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d9419dcae5c6468b8ddec76c29872c65 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:36:56.544704 32025 raft_consensus.cc:504] T 00000000000000000000000000000000 P d9419dcae5c6468b8ddec76c29872c65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9419dcae5c6468b8ddec76c29872c65" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 36583 } }
I20260620 18:36:56.545207 32025 leader_election.cc:302] T 00000000000000000000000000000000 P d9419dcae5c6468b8ddec76c29872c65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d9419dcae5c6468b8ddec76c29872c65; no voters:
I20260620 18:36:56.545789 32025 leader_election.cc:288] T 00000000000000000000000000000000 P d9419dcae5c6468b8ddec76c29872c65 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:36:56.545943 32030 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d9419dcae5c6468b8ddec76c29872c65 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:36:56.546614 32030 raft_consensus.cc:686] T 00000000000000000000000000000000 P d9419dcae5c6468b8ddec76c29872c65 [term 1 LEADER]: Becoming Leader. State: Replica: d9419dcae5c6468b8ddec76c29872c65, State: Running, Role: LEADER
I20260620 18:36:56.547113 32030 consensus_queue.cc:237] T 00000000000000000000000000000000 P d9419dcae5c6468b8ddec76c29872c65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9419dcae5c6468b8ddec76c29872c65" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 36583 } }
I20260620 18:36:56.547910 32025 sys_catalog.cc:564] T 00000000000000000000000000000000 P d9419dcae5c6468b8ddec76c29872c65 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:36:56.550202 32032 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9419dcae5c6468b8ddec76c29872c65 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d9419dcae5c6468b8ddec76c29872c65. Latest consensus state: current_term: 1 leader_uuid: "d9419dcae5c6468b8ddec76c29872c65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9419dcae5c6468b8ddec76c29872c65" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 36583 } } }
I20260620 18:36:56.550565 32032 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9419dcae5c6468b8ddec76c29872c65 [sys.catalog]: This master's current role is: LEADER
I20260620 18:36:56.550307 32031 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9419dcae5c6468b8ddec76c29872c65 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d9419dcae5c6468b8ddec76c29872c65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9419dcae5c6468b8ddec76c29872c65" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 36583 } } }
I20260620 18:36:56.550967 32031 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9419dcae5c6468b8ddec76c29872c65 [sys.catalog]: This master's current role is: LEADER
I20260620 18:36:56.555420 32035 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:36:56.559908 32035 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:36:56.568354 32035 catalog_manager.cc:1347] Generated new cluster ID: 3dd937dccba74bfcb9c8ab4d67d116a4
I20260620 18:36:56.568472 32035 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:36:56.589629 32035 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:36:56.590883 32035 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:36:56.599639 32035 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d9419dcae5c6468b8ddec76c29872c65: Generated new TSK 0
I20260620 18:36:56.600436 32035 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:36:56.609776 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980527431848-22676-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980527431848-22676-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980527431848-22676-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980527431848-22676-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.37.1:0
--local_ip_for_outbound_sockets=127.22.37.1
--webserver_interface=127.22.37.1
--webserver_port=0
--tserver_master_addrs=127.22.37.62:36583
--builtin_ntp_servers=127.22.37.20:32873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260620 18:36:56.939992 32049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:36:56.940281 32049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:36:56.940369 32049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:36:56.947435 32049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:36:56.947546 32049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:36:56.947650 32049 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.37.1
I20260620 18:36:56.957203 32049 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:32873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980527431848-22676-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980527431848-22676-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.37.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980527431848-22676-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.37.1
--webserver_port=0
--tserver_master_addrs=127.22.37.62:36583
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.37.1
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980527431848-22676-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:36:56.958570 32049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:36:56.960584 32049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:36:56.973839 32054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:56.973876 32057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:56.973905 32055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:56.974371 32049 server_base.cc:1034] running on GCE node
I20260620 18:36:56.975339 32049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:36:56.976210 32049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:36:56.977461 32049 hybrid_clock.cc:648] HybridClock initialized: now 1781980616977403 us; error 43 us; skew 500 ppm
I20260620 18:36:56.977747 32049 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:36:56.979988 32049 webserver.cc:460] Webserver started at http://127.22.37.1:38583/ using document root <none> and password file <none>
I20260620 18:36:56.980643 32049 fs_manager.cc:359] Metadata directory not provided
I20260620 18:36:56.980727 32049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:36:56.981006 32049 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:36:56.983862 32049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980527431848-22676-0/minicluster-data/ts-0/data/instance:
uuid: "648826393c414e5b83bdb524f0a5fa57"
format_stamp: "Formatted at 2026-06-20 18:36:56 on dist-test-slave-5lgp"
server_key: "da7a8637adac56384d6146dd66a35e6a"
server_key_iv: "97925a8ed735326b2316210838d0bb03"
server_key_version: "encryptionkey@0"
I20260620 18:36:56.984617 32049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980527431848-22676-0/minicluster-data/ts-0/wal/instance:
uuid: "648826393c414e5b83bdb524f0a5fa57"
format_stamp: "Formatted at 2026-06-20 18:36:56 on dist-test-slave-5lgp"
server_key: "da7a8637adac56384d6146dd66a35e6a"
server_key_iv: "97925a8ed735326b2316210838d0bb03"
server_key_version: "encryptionkey@0"
I20260620 18:36:56.989768 32049 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 18:36:56.993297 32063 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:56.994743 32049 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:36:56.994906 32049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980527431848-22676-0/minicluster-data/ts-0/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980527431848-22676-0/minicluster-data/ts-0/wal
uuid: "648826393c414e5b83bdb524f0a5fa57"
format_stamp: "Formatted at 2026-06-20 18:36:56 on dist-test-slave-5lgp"
server_key: "da7a8637adac56384d6146dd66a35e6a"
server_key_iv: "97925a8ed735326b2316210838d0bb03"
server_key_version: "encryptionkey@0"
I20260620 18:36:56.995083 32049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980527431848-22676-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980527431848-22676-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980527431848-22676-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:36:57.016311 32049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:36:57.017151 32049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:36:57.017452 32049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:36:57.018425 32049 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:36:57.019778 32049 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:36:57.019861 32049 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:57.019950 32049 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:36:57.020004 32049 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:57.053203 32049 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.1:36549
I20260620 18:36:57.053306 32175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.1:36549 every 8 connection(s)
I20260620 18:36:57.054678 32049 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980527431848-22676-0/minicluster-data/ts-0/data/info.pb
I20260620 18:36:57.061878 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 32049
I20260620 18:36:57.062109 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980527431848-22676-0/minicluster-data/ts-0/wal/instance
I20260620 18:36:57.062758 22676 external_mini_cluster.cc:1428] Setting key f050ac1d87867c12674b6cf74c897440
I20260620 18:36:57.065326 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980527431848-22676-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980527431848-22676-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980527431848-22676-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980527431848-22676-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.37.2:0
--local_ip_for_outbound_sockets=127.22.37.2
--webserver_interface=127.22.37.2
--webserver_port=0
--tserver_master_addrs=127.22.37.62:36583
--builtin_ntp_servers=127.22.37.20:32873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 18:36:57.067765 32178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:57.057324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.1:54159 (local address 127.22.37.62:36583)
0620 18:36:57.057765 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:57.057770 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:57.059095 (+ 1325us) server_negotiation.cc:407] Connection header received
0620 18:36:57.061029 (+ 1934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:57.061034 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:57.061135 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:57.061253 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:57.062392 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:57.063163 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:57.064966 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:57.065434 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:57.065567 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:57.066962 (+ 1395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:57.066967 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:57.066970 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:36:57.066987 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:57.067031 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:36:57.067043 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:57.067142 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:57.067354 (+ 212us) server_negotiation.cc:299] Negotiation successful
0620 18:36:57.067444 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":216,"threads_started":1}
I20260620 18:36:57.068778 32179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:57.057995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:36583 (local address 127.22.37.1:54159)
0620 18:36:57.058927 (+ 932us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:57.058973 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:57.060735 (+ 1762us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:57.061497 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:57.061506 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:57.061593 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:57.062228 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:57.062255 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:57.063313 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:57.063318 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:57.064827 (+ 1509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:57.064838 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:57.065653 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:57.065663 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:57.065888 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:57.066629 (+ 741us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:36:57.066632 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:57.066671 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:36:57.066748 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:36:57.067140 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:36:57.067152 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:36:57.067162 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 18:36:57.067347 (+ 185us) client_negotiation.cc:240] Negotiation successful
0620 18:36:57.067738 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":252,"threads_started":1}
I20260620 18:36:57.070335 32176 heartbeater.cc:347] Connected to a master server at 127.22.37.62:36583
I20260620 18:36:57.070592 32176 heartbeater.cc:464] Registering TS with master...
I20260620 18:36:57.071285 32176 heartbeater.cc:511] Master 127.22.37.62:36583 requested a full tablet report, sending...
I20260620 18:36:57.073261 31990 ts_manager.cc:194] Registered new tserver with Master: 648826393c414e5b83bdb524f0a5fa57 (127.22.37.1:36549)
I20260620 18:36:57.074924 31990 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.37.1:54159
W20260620 18:36:57.390630 32180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:36:57.390889 32180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:36:57.391042 32180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:36:57.398319 32180 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:36:57.398442 32180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:36:57.398630 32180 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.37.2
I20260620 18:36:57.407725 32180 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:32873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980527431848-22676-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980527431848-22676-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.37.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980527431848-22676-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.37.2
--webserver_port=0
--tserver_master_addrs=127.22.37.62:36583
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.37.2
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980527431848-22676-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:36:57.408972 32180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:36:57.410734 32180 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:36:57.423906 32186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:57.423928 32188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:57.424713 32185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:57.424732 32180 server_base.cc:1034] running on GCE node
I20260620 18:36:57.425658 32180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:36:57.426775 32180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:36:57.428038 32180 hybrid_clock.cc:648] HybridClock initialized: now 1781980617427974 us; error 50 us; skew 500 ppm
I20260620 18:36:57.428443 32180 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:36:57.430961 32180 webserver.cc:460] Webserver started at http://127.22.37.2:39305/ using document root <none> and password file <none>
I20260620 18:36:57.431751 32180 fs_manager.cc:359] Metadata directory not provided
I20260620 18:36:57.431896 32180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:36:57.432265 32180 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:36:57.435428 32180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980527431848-22676-0/minicluster-data/ts-1/data/instance:
uuid: "00cd111e777746c6aae67ccc84d52055"
format_stamp: "Formatted at 2026-06-20 18:36:57 on dist-test-slave-5lgp"
server_key: "ea50aba6b282e2cf6e5ab6abd0724dbf"
server_key_iv: "123990b537e3dbc97f6e8c7084d0e9d8"
server_key_version: "encryptionkey@0"
I20260620 18:36:57.436290 32180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980527431848-22676-0/minicluster-data/ts-1/wal/instance:
uuid: "00cd111e777746c6aae67ccc84d52055"
format_stamp: "Formatted at 2026-06-20 18:36:57 on dist-test-slave-5lgp"
server_key: "ea50aba6b282e2cf6e5ab6abd0724dbf"
server_key_iv: "123990b537e3dbc97f6e8c7084d0e9d8"
server_key_version: "encryptionkey@0"
I20260620 18:36:57.441982 32180 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 18:36:57.445655 32194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:57.446995 32180 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:36:57.447178 32180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980527431848-22676-0/minicluster-data/ts-1/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980527431848-22676-0/minicluster-data/ts-1/wal
uuid: "00cd111e777746c6aae67ccc84d52055"
format_stamp: "Formatted at 2026-06-20 18:36:57 on dist-test-slave-5lgp"
server_key: "ea50aba6b282e2cf6e5ab6abd0724dbf"
server_key_iv: "123990b537e3dbc97f6e8c7084d0e9d8"
server_key_version: "encryptionkey@0"
I20260620 18:36:57.447400 32180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980527431848-22676-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980527431848-22676-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980527431848-22676-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:36:57.484540 32180 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:36:57.485476 32180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:36:57.485764 32180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:36:57.486819 32180 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:36:57.488245 32180 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:36:57.488368 32180 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:57.488507 32180 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:36:57.488595 32180 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:57.523371 32180 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.2:33895
I20260620 18:36:57.523428 32306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.2:33895 every 8 connection(s)
I20260620 18:36:57.524889 32180 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980527431848-22676-0/minicluster-data/ts-1/data/info.pb
I20260620 18:36:57.528272 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 32180
I20260620 18:36:57.528455 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980527431848-22676-0/minicluster-data/ts-1/wal/instance
I20260620 18:36:57.529058 22676 external_mini_cluster.cc:1428] Setting key c07a818c98a8c8e544709c81fa586795
I20260620 18:36:57.531728 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980527431848-22676-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980527431848-22676-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980527431848-22676-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980527431848-22676-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.37.3:0
--local_ip_for_outbound_sockets=127.22.37.3
--webserver_interface=127.22.37.3
--webserver_port=0
--tserver_master_addrs=127.22.37.62:36583
--builtin_ntp_servers=127.22.37.20:32873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 18:36:57.538952 32178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:57.527561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.2:55725 (local address 127.22.37.62:36583)
0620 18:36:57.527784 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:57.527791 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:57.529574 (+ 1783us) server_negotiation.cc:407] Connection header received
0620 18:36:57.532070 (+ 2496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:57.532075 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:57.532165 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:57.532232 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:57.533815 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:57.534548 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:57.536124 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:57.536519 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:57.536654 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:57.538094 (+ 1440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:57.538101 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:57.538106 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:36:57.538133 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:57.538162 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:36:57.538174 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:57.538272 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:57.538540 (+ 268us) server_negotiation.cc:299] Negotiation successful
0620 18:36:57.538634 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 18:36:57.539984 32309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:57.528184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:36583 (local address 127.22.37.2:55725)
0620 18:36:57.529417 (+ 1233us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:57.529469 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:57.531763 (+ 2294us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:57.532707 (+ 944us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:57.532718 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:57.532841 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:57.533644 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:57.533664 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:57.534708 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:57.534713 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:57.535957 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:57.535986 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:57.536739 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:57.536745 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:57.537006 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:57.537762 (+ 756us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:36:57.537764 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:57.537802 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:36:57.537881 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:36:57.538272 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:36:57.538284 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:36:57.538304 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 18:36:57.538521 (+ 217us) client_negotiation.cc:240] Negotiation successful
0620 18:36:57.538943 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":354,"threads_started":1}
I20260620 18:36:57.541448 32307 heartbeater.cc:347] Connected to a master server at 127.22.37.62:36583
I20260620 18:36:57.541765 32307 heartbeater.cc:464] Registering TS with master...
I20260620 18:36:57.542704 32307 heartbeater.cc:511] Master 127.22.37.62:36583 requested a full tablet report, sending...
I20260620 18:36:57.544235 31990 ts_manager.cc:194] Registered new tserver with Master: 00cd111e777746c6aae67ccc84d52055 (127.22.37.2:33895)
I20260620 18:36:57.545459 31990 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.37.2:55725
W20260620 18:36:57.854419 32310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:36:57.854677 32310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:36:57.854756 32310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:36:57.861865 32310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:36:57.861958 32310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:36:57.862061 32310 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.37.3
I20260620 18:36:57.870839 32310 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:32873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980527431848-22676-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980527431848-22676-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.37.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980527431848-22676-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.37.3
--webserver_port=0
--tserver_master_addrs=127.22.37.62:36583
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.37.3
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980527431848-22676-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:36:57.871955 32310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:36:57.873709 32310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:36:57.886538 32316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:57.886508 32318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:57.887224 32315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:57.887313 32310 server_base.cc:1034] running on GCE node
I20260620 18:36:57.887897 32310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:36:57.888852 32310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:36:57.890069 32310 hybrid_clock.cc:648] HybridClock initialized: now 1781980617890012 us; error 50 us; skew 500 ppm
I20260620 18:36:57.890393 32310 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:36:57.892589 32310 webserver.cc:460] Webserver started at http://127.22.37.3:36917/ using document root <none> and password file <none>
I20260620 18:36:57.893283 32310 fs_manager.cc:359] Metadata directory not provided
I20260620 18:36:57.893437 32310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:36:57.893775 32310 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:36:57.896535 32310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980527431848-22676-0/minicluster-data/ts-2/data/instance:
uuid: "d050cd91074540ed966b126f879919b4"
format_stamp: "Formatted at 2026-06-20 18:36:57 on dist-test-slave-5lgp"
server_key: "1f9927be2712095adf462bdedacf1469"
server_key_iv: "f221ebbe68603e974d235a34df320582"
server_key_version: "encryptionkey@0"
I20260620 18:36:57.897294 32310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980527431848-22676-0/minicluster-data/ts-2/wal/instance:
uuid: "d050cd91074540ed966b126f879919b4"
format_stamp: "Formatted at 2026-06-20 18:36:57 on dist-test-slave-5lgp"
server_key: "1f9927be2712095adf462bdedacf1469"
server_key_iv: "f221ebbe68603e974d235a34df320582"
server_key_version: "encryptionkey@0"
I20260620 18:36:57.902348 32310 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260620 18:36:57.905624 32324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:57.906810 32310 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 18:36:57.906984 32310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980527431848-22676-0/minicluster-data/ts-2/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980527431848-22676-0/minicluster-data/ts-2/wal
uuid: "d050cd91074540ed966b126f879919b4"
format_stamp: "Formatted at 2026-06-20 18:36:57 on dist-test-slave-5lgp"
server_key: "1f9927be2712095adf462bdedacf1469"
server_key_iv: "f221ebbe68603e974d235a34df320582"
server_key_version: "encryptionkey@0"
I20260620 18:36:57.907189 32310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980527431848-22676-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980527431848-22676-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980527431848-22676-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:36:57.926864 32310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:36:57.927666 32310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:36:57.927937 32310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:36:57.928893 32310 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:36:57.930331 32310 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:36:57.930452 32310 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:57.930624 32310 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:36:57.930714 32310 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:57.963872 32310 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.3:39317
I20260620 18:36:57.963960 32436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.3:39317 every 8 connection(s)
I20260620 18:36:57.965159 32310 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980527431848-22676-0/minicluster-data/ts-2/data/info.pb
I20260620 18:36:57.975227 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 32310
I20260620 18:36:57.975405 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980527431848-22676-0/minicluster-data/ts-2/wal/instance
I20260620 18:36:57.975884 22676 external_mini_cluster.cc:1428] Setting key 35b30d940d382370f56c01f4f0e53e43
I20260620 18:36:57.980562 32178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:57.967754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.3:53823 (local address 127.22.37.62:36583)
0620 18:36:57.967943 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:57.967947 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:57.969300 (+ 1353us) server_negotiation.cc:407] Connection header received
0620 18:36:57.971140 (+ 1840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:57.971146 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:57.971233 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:57.971358 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:57.972402 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:57.973130 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:57.974852 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:57.975378 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:57.975493 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:57.979762 (+ 4269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:57.979768 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:57.979774 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:36:57.979797 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:57.979827 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:36:57.979839 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:57.979935 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:57.980156 (+ 221us) server_negotiation.cc:299] Negotiation successful
0620 18:36:57.980251 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 18:36:57.981683 32439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:57.968279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:36583 (local address 127.22.37.3:53823)
0620 18:36:57.969144 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:57.969186 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:57.970874 (+ 1688us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:57.971589 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:57.971597 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:57.971683 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:57.972237 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:57.972253 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:57.973280 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:57.973288 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:57.974680 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:57.974700 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:57.975599 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:57.975607 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:57.975818 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:57.979436 (+ 3618us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:36:57.979438 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:57.979479 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:36:57.979571 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:36:57.979934 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:36:57.979946 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:36:57.979956 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 18:36:57.980135 (+ 179us) client_negotiation.cc:240] Negotiation successful
0620 18:36:57.980545 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":254,"threads_started":1}
I20260620 18:36:57.982873 32437 heartbeater.cc:347] Connected to a master server at 127.22.37.62:36583
I20260620 18:36:57.983150 32437 heartbeater.cc:464] Registering TS with master...
I20260620 18:36:57.983857 32437 heartbeater.cc:511] Master 127.22.37.62:36583 requested a full tablet report, sending...
I20260620 18:36:57.985224 31990 ts_manager.cc:194] Registered new tserver with Master: d050cd91074540ed966b126f879919b4 (127.22.37.3:39317)
I20260620 18:36:57.986058 31990 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.37.3:53823
I20260620 18:36:57.991014 22676 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 18:36:57.997030 32178 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:36:57.995329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49962 (local address 127.22.37.62:36583)
0620 18:36:57.995586 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:57.995595 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:57.995974 (+ 379us) server_negotiation.cc:407] Connection header received
0620 18:36:57.996192 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:57.996197 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:57.996281 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:57.996384 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:57.996776 (+ 392us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49962: BlockingRecv error: recv got EOF from 127.0.0.1:49962 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 18:36:57.997769 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 32049
I20260620 18:36:58.005883 32446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:57.998727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44012 (local address 127.22.37.1:36549)
0620 18:36:57.999203 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:57.999214 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:57.999246 (+ 32us) server_negotiation.cc:407] Connection header received
0620 18:36:57.999318 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:57.999324 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:57.999538 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:57.999709 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:58.000139 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:58.001103 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:58.002414 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:58.003009 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:58.003147 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:58.004313 (+ 1166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:58.004322 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:58.004356 (+ 34us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:36:58.005071 (+ 715us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:58.005122 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:36:58.005141 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:58.005260 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:58.005403 (+ 143us) server_negotiation.cc:299] Negotiation successful
0620 18:36:58.005553 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":215,"threads_started":1}
I20260620 18:36:58.007124 32171 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:36:58.094087 32176 heartbeater.cc:503] Master 127.22.37.62:36583 was elected leader, sending a full tablet report...
I20260620 18:36:58.097481 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 32049
I20260620 18:36:58.117820 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 32180
I20260620 18:36:58.128437 32448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:58.118864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34978 (local address 127.22.37.2:33895)
0620 18:36:58.119390 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:58.119397 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:58.119430 (+ 33us) server_negotiation.cc:407] Connection header received
0620 18:36:58.119489 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:58.119495 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:58.119709 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:58.119832 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:58.120433 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:58.121514 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:58.123666 (+ 2152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:58.124677 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:58.124833 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:58.126311 (+ 1478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:58.126325 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:58.126346 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:36:58.127304 (+ 958us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:58.127391 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:36:58.127419 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:58.127539 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:58.127747 (+ 208us) server_negotiation.cc:299] Negotiation successful
0620 18:36:58.127914 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":265,"threads_started":1}
I20260620 18:36:58.130069 32302 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:36:58.211655 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 32180
I20260620 18:36:58.228103 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 32310
I20260620 18:36:58.236083 32450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:58.229030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34570 (local address 127.22.37.3:39317)
0620 18:36:58.229735 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:58.229742 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:58.229773 (+ 31us) server_negotiation.cc:407] Connection header received
0620 18:36:58.229831 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:58.229837 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:58.230063 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:58.230186 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:58.230630 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:58.231733 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:58.232947 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:58.233653 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:58.233746 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:58.234750 (+ 1004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:58.234757 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:58.234770 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:36:58.235347 (+ 577us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:58.235389 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:36:58.235407 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:58.235490 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:58.235662 (+ 172us) server_negotiation.cc:299] Negotiation successful
0620 18:36:58.235761 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":409,"threads_started":1}
I20260620 18:36:58.237347 32432 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:36:58.319885 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 32310
I20260620 18:36:58.335666 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 31959
I20260620 18:36:58.337030 32020 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:36:58.438570 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 31959
2026-06-20T18:36:58Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2419 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-20T18:36:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:36:58Z Disabled control of system clock
I20260620 18:36:58.507601 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980527431848-22676-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980527431848-22676-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980527431848-22676-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.62:41079
--webserver_interface=127.22.37.62
--webserver_port=0
--builtin_ntp_servers=127.22.37.20:43599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.37.62:41079
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 18:36:58.828262 32456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:36:58.828539 32456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:36:58.828696 32456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:36:58.835851 32456 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:36:58.835943 32456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:36:58.836050 32456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:36:58.836155 32456 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:36:58.836237 32456 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:36:58.844990 32456 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:43599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980527431848-22676-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980527431848-22676-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.37.62:41079
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.62:41079
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.37.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980527431848-22676-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:36:58.846393 32456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:36:58.848126 32456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:36:58.859592 32462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:58.859619 32461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:58.860173 32464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:58.860416 32456 server_base.cc:1034] running on GCE node
I20260620 18:36:58.861239 32456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:36:58.862653 32456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:36:58.863893 32456 hybrid_clock.cc:648] HybridClock initialized: now 1781980618863848 us; error 50 us; skew 500 ppm
I20260620 18:36:58.864238 32456 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:36:58.866395 32456 webserver.cc:460] Webserver started at http://127.22.37.62:45091/ using document root <none> and password file <none>
I20260620 18:36:58.867107 32456 fs_manager.cc:359] Metadata directory not provided
I20260620 18:36:58.867236 32456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:36:58.867552 32456 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:36:58.870448 32456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980527431848-22676-0/minicluster-data/master-0/data/instance:
uuid: "987716804bc143a490c860154ae42643"
format_stamp: "Formatted at 2026-06-20 18:36:58 on dist-test-slave-5lgp"
server_key: "e2ed4c485b7223e79760f2694764ea12"
server_key_iv: "4bbf537534a6a1e8e5ff8492c9994a96"
server_key_version: "encryptionkey@0"
I20260620 18:36:58.871235 32456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980527431848-22676-0/minicluster-data/master-0/wal/instance:
uuid: "987716804bc143a490c860154ae42643"
format_stamp: "Formatted at 2026-06-20 18:36:58 on dist-test-slave-5lgp"
server_key: "e2ed4c485b7223e79760f2694764ea12"
server_key_iv: "4bbf537534a6a1e8e5ff8492c9994a96"
server_key_version: "encryptionkey@0"
I20260620 18:36:58.876258 32456 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 18:36:58.879690 32470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:58.880991 32456 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260620 18:36:58.881174 32456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980527431848-22676-0/minicluster-data/master-0/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980527431848-22676-0/minicluster-data/master-0/wal
uuid: "987716804bc143a490c860154ae42643"
format_stamp: "Formatted at 2026-06-20 18:36:58 on dist-test-slave-5lgp"
server_key: "e2ed4c485b7223e79760f2694764ea12"
server_key_iv: "4bbf537534a6a1e8e5ff8492c9994a96"
server_key_version: "encryptionkey@0"
I20260620 18:36:58.881402 32456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980527431848-22676-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980527431848-22676-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980527431848-22676-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:36:58.886840 32456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:36:58.887564 32456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:36:58.887863 32456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:36:58.906867 32456 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.62:41079
I20260620 18:36:58.906884 32521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.62:41079 every 8 connection(s)
I20260620 18:36:58.908259 32456 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
I20260620 18:36:58.911983 32522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:36:58.917407 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 32456
I20260620 18:36:58.917658 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980527431848-22676-0/minicluster-data/master-0/wal/instance
I20260620 18:36:58.918272 22676 external_mini_cluster.cc:1428] Setting key c8c76662715809cdbd4ad8436d4ec038
I20260620 18:36:58.923658 32522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 987716804bc143a490c860154ae42643: Bootstrap starting.
I20260620 18:36:58.927526 32522 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 987716804bc143a490c860154ae42643: Neither blocks nor log segments found. Creating new log.
I20260620 18:36:58.927740 32525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:58.920390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52964 (local address 127.22.37.62:41079)
0620 18:36:58.921557 (+ 1167us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:58.921572 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:58.921620 (+ 48us) server_negotiation.cc:407] Connection header received
0620 18:36:58.923854 (+ 2234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:58.923881 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:58.924380 (+ 499us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:58.924793 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:58.925863 (+ 1070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:58.925873 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:58.925899 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:36:58.925917 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:58.925986 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:36:58.926011 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:58.926209 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:58.926331 (+ 122us) server_negotiation.cc:299] Negotiation successful
0620 18:36:58.926587 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":390,"threads_started":1}
I20260620 18:36:58.928665 32522 log.cc:824] T 00000000000000000000000000000000 P 987716804bc143a490c860154ae42643: Log is configured to *not* fsync() on all Append() calls
I20260620 18:36:58.931496 32522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 987716804bc143a490c860154ae42643: No bootstrap required, opened a new log
I20260620 18:36:58.938727 32522 raft_consensus.cc:348] T 00000000000000000000000000000000 P 987716804bc143a490c860154ae42643 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "987716804bc143a490c860154ae42643" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 41079 } }
I20260620 18:36:58.939117 32522 raft_consensus.cc:374] T 00000000000000000000000000000000 P 987716804bc143a490c860154ae42643 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:36:58.939221 32522 raft_consensus.cc:729] T 00000000000000000000000000000000 P 987716804bc143a490c860154ae42643 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 987716804bc143a490c860154ae42643, State: Initialized, Role: FOLLOWER
I20260620 18:36:58.939800 32522 consensus_queue.cc:260] T 00000000000000000000000000000000 P 987716804bc143a490c860154ae42643 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "987716804bc143a490c860154ae42643" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 41079 } }
I20260620 18:36:58.940035 32522 raft_consensus.cc:388] T 00000000000000000000000000000000 P 987716804bc143a490c860154ae42643 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:36:58.940188 32522 raft_consensus.cc:482] T 00000000000000000000000000000000 P 987716804bc143a490c860154ae42643 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:36:58.940371 32522 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 987716804bc143a490c860154ae42643 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:36:58.942588 32522 raft_consensus.cc:504] T 00000000000000000000000000000000 P 987716804bc143a490c860154ae42643 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "987716804bc143a490c860154ae42643" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 41079 } }
I20260620 18:36:58.943100 32522 leader_election.cc:302] T 00000000000000000000000000000000 P 987716804bc143a490c860154ae42643 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 987716804bc143a490c860154ae42643; no voters:
I20260620 18:36:58.943646 32522 leader_election.cc:288] T 00000000000000000000000000000000 P 987716804bc143a490c860154ae42643 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:36:58.943775 32527 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 987716804bc143a490c860154ae42643 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:36:58.944855 32527 raft_consensus.cc:686] T 00000000000000000000000000000000 P 987716804bc143a490c860154ae42643 [term 1 LEADER]: Becoming Leader. State: Replica: 987716804bc143a490c860154ae42643, State: Running, Role: LEADER
I20260620 18:36:58.945504 32527 consensus_queue.cc:237] T 00000000000000000000000000000000 P 987716804bc143a490c860154ae42643 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "987716804bc143a490c860154ae42643" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 41079 } }
I20260620 18:36:58.945840 32522 sys_catalog.cc:564] T 00000000000000000000000000000000 P 987716804bc143a490c860154ae42643 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:36:58.948207 32528 sys_catalog.cc:455] T 00000000000000000000000000000000 P 987716804bc143a490c860154ae42643 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "987716804bc143a490c860154ae42643" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "987716804bc143a490c860154ae42643" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 41079 } } }
I20260620 18:36:58.948546 32528 sys_catalog.cc:458] T 00000000000000000000000000000000 P 987716804bc143a490c860154ae42643 [sys.catalog]: This master's current role is: LEADER
I20260620 18:36:58.948294 32529 sys_catalog.cc:455] T 00000000000000000000000000000000 P 987716804bc143a490c860154ae42643 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 987716804bc143a490c860154ae42643. Latest consensus state: current_term: 1 leader_uuid: "987716804bc143a490c860154ae42643" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "987716804bc143a490c860154ae42643" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 41079 } } }
I20260620 18:36:58.948712 32529 sys_catalog.cc:458] T 00000000000000000000000000000000 P 987716804bc143a490c860154ae42643 [sys.catalog]: This master's current role is: LEADER
I20260620 18:36:58.950541 32533 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:36:58.956482 32533 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:36:58.965519 32533 catalog_manager.cc:1347] Generated new cluster ID: 05ee0e89e5b14016b18f6007d258d339
I20260620 18:36:58.965642 32533 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:36:58.975370 32533 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:36:58.975842 32533 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:36:58.982968 32533 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 987716804bc143a490c860154ae42643: Generated new TSK 0
I20260620 18:36:58.983875 32533 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:36:58.996111 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980527431848-22676-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980527431848-22676-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980527431848-22676-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980527431848-22676-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.37.1:0
--local_ip_for_outbound_sockets=127.22.37.1
--webserver_interface=127.22.37.1
--webserver_port=0
--tserver_master_addrs=127.22.37.62:41079
--builtin_ntp_servers=127.22.37.20:43599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 18:36:59.329437 32546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:36:59.329721 32546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:36:59.329890 32546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:36:59.336970 32546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:36:59.337085 32546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:36:59.337265 32546 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.37.1
I20260620 18:36:59.346913 32546 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:43599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980527431848-22676-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980527431848-22676-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.37.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980527431848-22676-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.37.1
--webserver_port=0
--tserver_master_addrs=127.22.37.62:41079
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.37.1
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980527431848-22676-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:36:59.348286 32546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:36:59.350204 32546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:36:59.363770 32551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:59.363696 32552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:59.363731 32554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:59.364512 32546 server_base.cc:1034] running on GCE node
I20260620 18:36:59.365074 32546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:36:59.366075 32546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:36:59.367259 32546 hybrid_clock.cc:648] HybridClock initialized: now 1781980619367210 us; error 58 us; skew 500 ppm
I20260620 18:36:59.367548 32546 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:36:59.369859 32546 webserver.cc:460] Webserver started at http://127.22.37.1:40869/ using document root <none> and password file <none>
I20260620 18:36:59.370554 32546 fs_manager.cc:359] Metadata directory not provided
I20260620 18:36:59.370668 32546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:36:59.371001 32546 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:36:59.373816 32546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980527431848-22676-0/minicluster-data/ts-0/data/instance:
uuid: "48e23945303647708eb21046e6e87562"
format_stamp: "Formatted at 2026-06-20 18:36:59 on dist-test-slave-5lgp"
server_key: "ff51d7b20180309e81acf800a406a6c7"
server_key_iv: "330548f84c1c2da4bc119e1177fc4ef0"
server_key_version: "encryptionkey@0"
I20260620 18:36:59.374593 32546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980527431848-22676-0/minicluster-data/ts-0/wal/instance:
uuid: "48e23945303647708eb21046e6e87562"
format_stamp: "Formatted at 2026-06-20 18:36:59 on dist-test-slave-5lgp"
server_key: "ff51d7b20180309e81acf800a406a6c7"
server_key_iv: "330548f84c1c2da4bc119e1177fc4ef0"
server_key_version: "encryptionkey@0"
I20260620 18:36:59.379689 32546 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260620 18:36:59.383155 32560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:59.384429 32546 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 18:36:59.384619 32546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980527431848-22676-0/minicluster-data/ts-0/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980527431848-22676-0/minicluster-data/ts-0/wal
uuid: "48e23945303647708eb21046e6e87562"
format_stamp: "Formatted at 2026-06-20 18:36:59 on dist-test-slave-5lgp"
server_key: "ff51d7b20180309e81acf800a406a6c7"
server_key_iv: "330548f84c1c2da4bc119e1177fc4ef0"
server_key_version: "encryptionkey@0"
I20260620 18:36:59.384833 32546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980527431848-22676-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980527431848-22676-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980527431848-22676-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:36:59.390302 32546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:36:59.391072 32546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:36:59.391340 32546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:36:59.392230 32546 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:36:59.393739 32546 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:36:59.393852 32546 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:59.393985 32546 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:36:59.394078 32546 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:59.425668 32546 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.1:45041
I20260620 18:36:59.425755 32672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.1:45041 every 8 connection(s)
I20260620 18:36:59.427184 32546 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980527431848-22676-0/minicluster-data/ts-0/data/info.pb
I20260620 18:36:59.430610 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 32546
I20260620 18:36:59.430826 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980527431848-22676-0/minicluster-data/ts-0/wal/instance
I20260620 18:36:59.431433 22676 external_mini_cluster.cc:1428] Setting key d57bfd982baa1ab4ab86d22a8e2c8ced
I20260620 18:36:59.434262 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980527431848-22676-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980527431848-22676-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980527431848-22676-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980527431848-22676-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.37.2:0
--local_ip_for_outbound_sockets=127.22.37.2
--webserver_interface=127.22.37.2
--webserver_port=0
--tserver_master_addrs=127.22.37.62:41079
--builtin_ntp_servers=127.22.37.20:43599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260620 18:36:59.436299 32675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:59.429898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.1:59753 (local address 127.22.37.62:41079)
0620 18:36:59.430506 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:59.430512 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:59.431986 (+ 1474us) server_negotiation.cc:407] Connection header received
0620 18:36:59.433905 (+ 1919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:59.433911 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:59.433996 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:59.434120 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:59.435476 (+ 1356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:59.435484 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:59.435491 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:36:59.435514 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:59.435566 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:36:59.435584 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:59.435707 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:59.435868 (+ 161us) server_negotiation.cc:299] Negotiation successful
0620 18:36:59.435934 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":379,"threads_started":1}
I20260620 18:36:59.437078 32676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:59.430623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:41079 (local address 127.22.37.1:59753)
0620 18:36:59.431788 (+ 1165us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:59.431834 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:59.433645 (+ 1811us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:59.434306 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:59.434313 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:59.434438 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:59.435183 (+ 745us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:36:59.435185 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:59.435223 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:36:59.435306 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:36:59.435678 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:36:59.435685 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:36:59.435690 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 18:36:59.435901 (+ 211us) client_negotiation.cc:240] Negotiation successful
0620 18:36:59.436098 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":380,"threads_started":1}
I20260620 18:36:59.438620 32673 heartbeater.cc:347] Connected to a master server at 127.22.37.62:41079
I20260620 18:36:59.438933 32673 heartbeater.cc:464] Registering TS with master...
I20260620 18:36:59.439589 32673 heartbeater.cc:511] Master 127.22.37.62:41079 requested a full tablet report, sending...
I20260620 18:36:59.441726 32487 ts_manager.cc:194] Registered new tserver with Master: 48e23945303647708eb21046e6e87562 (127.22.37.1:45041)
W20260620 18:36:59.758283 32677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:36:59.758535 32677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:36:59.758620 32677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:36:59.765489 32677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:36:59.765575 32677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:36:59.765682 32677 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.37.2
I20260620 18:36:59.774081 32677 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:43599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980527431848-22676-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980527431848-22676-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.37.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980527431848-22676-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.37.2
--webserver_port=0
--tserver_master_addrs=127.22.37.62:41079
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.37.2
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980527431848-22676-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:36:59.775272 32677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:36:59.776927 32677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:36:59.789548 32683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:59.789630 32682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:59.789557 32685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:59.790144 32677 server_base.cc:1034] running on GCE node
I20260620 18:36:59.790825 32677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:36:59.791723 32677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:36:59.792984 32677 hybrid_clock.cc:648] HybridClock initialized: now 1781980619792914 us; error 58 us; skew 500 ppm
I20260620 18:36:59.793314 32677 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:36:59.795538 32677 webserver.cc:460] Webserver started at http://127.22.37.2:38017/ using document root <none> and password file <none>
I20260620 18:36:59.796187 32677 fs_manager.cc:359] Metadata directory not provided
I20260620 18:36:59.796337 32677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:36:59.796651 32677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:36:59.799429 32677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980527431848-22676-0/minicluster-data/ts-1/data/instance:
uuid: "af736da7852249898eb29dfea59f9fb5"
format_stamp: "Formatted at 2026-06-20 18:36:59 on dist-test-slave-5lgp"
server_key: "2d5f0b3b5955f918bc83284048e59e2a"
server_key_iv: "a121ce428e7d0cbb7c7a81fc286198da"
server_key_version: "encryptionkey@0"
I20260620 18:36:59.800196 32677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980527431848-22676-0/minicluster-data/ts-1/wal/instance:
uuid: "af736da7852249898eb29dfea59f9fb5"
format_stamp: "Formatted at 2026-06-20 18:36:59 on dist-test-slave-5lgp"
server_key: "2d5f0b3b5955f918bc83284048e59e2a"
server_key_iv: "a121ce428e7d0cbb7c7a81fc286198da"
server_key_version: "encryptionkey@0"
I20260620 18:36:59.805087 32677 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260620 18:36:59.808498 32691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:59.809823 32677 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:36:59.809979 32677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980527431848-22676-0/minicluster-data/ts-1/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980527431848-22676-0/minicluster-data/ts-1/wal
uuid: "af736da7852249898eb29dfea59f9fb5"
format_stamp: "Formatted at 2026-06-20 18:36:59 on dist-test-slave-5lgp"
server_key: "2d5f0b3b5955f918bc83284048e59e2a"
server_key_iv: "a121ce428e7d0cbb7c7a81fc286198da"
server_key_version: "encryptionkey@0"
I20260620 18:36:59.810223 32677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980527431848-22676-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980527431848-22676-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980527431848-22676-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:36:59.815798 32677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:36:59.816483 32677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:36:59.816722 32677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:36:59.817625 32677 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:36:59.818935 32677 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:36:59.819021 32677 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:59.819159 32677 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:36:59.819234 32677 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:59.850174 32677 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.2:45521
I20260620 18:36:59.850239 335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.2:45521 every 8 connection(s)
I20260620 18:36:59.851529 32677 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980527431848-22676-0/minicluster-data/ts-1/data/info.pb
I20260620 18:36:59.856515 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 32677
I20260620 18:36:59.856703 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980527431848-22676-0/minicluster-data/ts-1/wal/instance
I20260620 18:36:59.857167 22676 external_mini_cluster.cc:1428] Setting key 07752111737fd33296a9026a62cfb400
I20260620 18:36:59.860176 32675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:59.854190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.2:60925 (local address 127.22.37.62:41079)
0620 18:36:59.854362 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:59.854366 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:59.855873 (+ 1507us) server_negotiation.cc:407] Connection header received
0620 18:36:59.857693 (+ 1820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:59.857698 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:59.857784 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:59.857891 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:59.859400 (+ 1509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:59.859405 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:59.859410 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:36:59.859429 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:59.859468 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:36:59.859484 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:59.859616 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:59.859773 (+ 157us) server_negotiation.cc:299] Negotiation successful
0620 18:36:59.859836 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 18:36:59.860198 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980527431848-22676-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980527431848-22676-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980527431848-22676-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980527431848-22676-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.37.3:0
--local_ip_for_outbound_sockets=127.22.37.3
--webserver_interface=127.22.37.3
--webserver_port=0
--tserver_master_addrs=127.22.37.62:41079
--builtin_ntp_servers=127.22.37.20:43599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260620 18:36:59.861053 338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:59.854720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:41079 (local address 127.22.37.2:60925)
0620 18:36:59.855704 (+ 984us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:59.855748 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:59.857416 (+ 1668us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:59.858144 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:59.858155 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:59.858278 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:59.859062 (+ 784us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:36:59.859065 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:59.859115 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:36:59.859226 (+ 111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:36:59.859582 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:36:59.859589 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:36:59.859594 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 18:36:59.859797 (+ 203us) client_negotiation.cc:240] Negotiation successful
0620 18:36:59.859999 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":246,"threads_started":1}
I20260620 18:36:59.862423 336 heartbeater.cc:347] Connected to a master server at 127.22.37.62:41079
I20260620 18:36:59.862718 336 heartbeater.cc:464] Registering TS with master...
I20260620 18:36:59.863276 336 heartbeater.cc:511] Master 127.22.37.62:41079 requested a full tablet report, sending...
I20260620 18:36:59.864575 32487 ts_manager.cc:194] Registered new tserver with Master: af736da7852249898eb29dfea59f9fb5 (127.22.37.2:45521)
W20260620 18:37:00.182046 339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:37:00.182297 339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:37:00.182382 339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:37:00.189531 339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:37:00.189615 339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:37:00.189715 339 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.37.3
I20260620 18:37:00.198392 339 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:43599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980527431848-22676-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980527431848-22676-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.37.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980527431848-22676-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.37.3
--webserver_port=0
--tserver_master_addrs=127.22.37.62:41079
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.37.3
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980527431848-22676-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:37:00.199539 339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:37:00.201143 339 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:37:00.213708 345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:00.213738 344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:00.213765 347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:37:00.214512 339 server_base.cc:1034] running on GCE node
I20260620 18:37:00.215087 339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:37:00.215888 339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:37:00.217106 339 hybrid_clock.cc:648] HybridClock initialized: now 1781980620217066 us; error 35 us; skew 500 ppm
I20260620 18:37:00.217458 339 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:37:00.224315 339 webserver.cc:460] Webserver started at http://127.22.37.3:42817/ using document root <none> and password file <none>
I20260620 18:37:00.225039 339 fs_manager.cc:359] Metadata directory not provided
I20260620 18:37:00.225176 339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:37:00.225576 339 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:37:00.228317 339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980527431848-22676-0/minicluster-data/ts-2/data/instance:
uuid: "b82bce5c1b294fa69a293577d1f2c6ab"
format_stamp: "Formatted at 2026-06-20 18:37:00 on dist-test-slave-5lgp"
server_key: "febebe73a1b62430d91d6fd234b89987"
server_key_iv: "e84ae7baff31edb5318f684152a8b41f"
server_key_version: "encryptionkey@0"
I20260620 18:37:00.229163 339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980527431848-22676-0/minicluster-data/ts-2/wal/instance:
uuid: "b82bce5c1b294fa69a293577d1f2c6ab"
format_stamp: "Formatted at 2026-06-20 18:37:00 on dist-test-slave-5lgp"
server_key: "febebe73a1b62430d91d6fd234b89987"
server_key_iv: "e84ae7baff31edb5318f684152a8b41f"
server_key_version: "encryptionkey@0"
I20260620 18:37:00.234112 339 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 18:37:00.237231 353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:37:00.238497 339 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260620 18:37:00.238682 339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980527431848-22676-0/minicluster-data/ts-2/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980527431848-22676-0/minicluster-data/ts-2/wal
uuid: "b82bce5c1b294fa69a293577d1f2c6ab"
format_stamp: "Formatted at 2026-06-20 18:37:00 on dist-test-slave-5lgp"
server_key: "febebe73a1b62430d91d6fd234b89987"
server_key_iv: "e84ae7baff31edb5318f684152a8b41f"
server_key_version: "encryptionkey@0"
I20260620 18:37:00.238904 339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980527431848-22676-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980527431848-22676-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980527431848-22676-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:37:00.244217 339 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:37:00.244931 339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:37:00.245184 339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:37:00.246145 339 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:37:00.247514 339 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:37:00.247639 339 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:37:00.247771 339 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:37:00.247875 339 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:37:00.280014 339 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.3:39409
I20260620 18:37:00.280099 465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.3:39409 every 8 connection(s)
I20260620 18:37:00.281286 339 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980527431848-22676-0/minicluster-data/ts-2/data/info.pb
I20260620 18:37:00.289204 32675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:00.283828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.3:35319 (local address 127.22.37.62:41079)
0620 18:37:00.284058 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:00.284065 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:00.285560 (+ 1495us) server_negotiation.cc:407] Connection header received
0620 18:37:00.287188 (+ 1628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:00.287192 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:00.287240 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:00.287328 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:00.288542 (+ 1214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:00.288546 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:00.288549 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:37:00.288584 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:00.288640 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:37:00.288662 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:00.288753 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:00.288902 (+ 149us) server_negotiation.cc:299] Negotiation successful
0620 18:37:00.288949 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 18:37:00.290143 468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:00.284418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:41079 (local address 127.22.37.3:35319)
0620 18:37:00.285427 (+ 1009us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:00.285463 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 18:37:00.286954 (+ 1491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:37:00.287514 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:37:00.287531 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:37:00.287618 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:37:00.288267 (+ 649us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:37:00.288269 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:37:00.288304 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:37:00.288382 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:37:00.288743 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:37:00.288749 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:37:00.288754 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 18:37:00.288945 (+ 191us) client_negotiation.cc:240] Negotiation successful
0620 18:37:00.289120 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":246,"threads_started":1}
I20260620 18:37:00.291501 466 heartbeater.cc:347] Connected to a master server at 127.22.37.62:41079
I20260620 18:37:00.291591 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 339
I20260620 18:37:00.291781 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980527431848-22676-0/minicluster-data/ts-2/wal/instance
I20260620 18:37:00.291798 466 heartbeater.cc:464] Registering TS with master...
I20260620 18:37:00.292227 22676 external_mini_cluster.cc:1428] Setting key d49494598b9c0e1af33745f81e92b3ad
I20260620 18:37:00.292366 466 heartbeater.cc:511] Master 127.22.37.62:41079 requested a full tablet report, sending...
I20260620 18:37:00.293758 32487 ts_manager.cc:194] Registered new tserver with Master: b82bce5c1b294fa69a293577d1f2c6ab (127.22.37.3:39409)
I20260620 18:37:00.295240 22676 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 18:37:00.301041 32675 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:37:00.299636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52976 (local address 127.22.37.62:41079)
0620 18:37:00.299820 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:00.299828 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:00.300291 (+ 463us) server_negotiation.cc:407] Connection header received
0620 18:37:00.300346 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:00.300350 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:00.300392 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:00.300483 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:00.300802 (+ 319us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52976: BlockingRecv error: recv got EOF from 127.0.0.1:52976 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 18:37:00.301821 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 32546
I20260620 18:37:00.305136 475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:00.302810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36598 (local address 127.22.37.1:45041)
0620 18:37:00.303317 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:00.303323 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:00.303347 (+ 24us) server_negotiation.cc:407] Connection header received
0620 18:37:00.303407 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:00.303413 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:00.303594 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:00.303708 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:00.304571 (+ 863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:00.304578 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:00.304588 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:37:00.304600 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:00.304640 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:37:00.304654 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:00.304755 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:00.304839 (+ 84us) server_negotiation.cc:299] Negotiation successful
0620 18:37:00.304911 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":249,"threads_started":1}
I20260620 18:37:00.306277 32668 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:37:00.386934 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 32546
I20260620 18:37:00.403580 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 32677
I20260620 18:37:00.407138 477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:00.404695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57000 (local address 127.22.37.2:45521)
0620 18:37:00.405147 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:00.405154 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:00.405180 (+ 26us) server_negotiation.cc:407] Connection header received
0620 18:37:00.405238 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:00.405244 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:00.405538 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:00.405665 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:00.406583 (+ 918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:00.406590 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:00.406602 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:37:00.406614 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:00.406659 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:37:00.406675 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:00.406771 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:00.406847 (+ 76us) server_negotiation.cc:299] Negotiation successful
0620 18:37:00.406925 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":203,"threads_started":1}
I20260620 18:37:00.408301 331 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:37:00.490506 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 32677
I20260620 18:37:00.506188 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 339
I20260620 18:37:00.509883 479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:00.507063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53016 (local address 127.22.37.3:39409)
0620 18:37:00.507658 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:00.507666 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:00.507695 (+ 29us) server_negotiation.cc:407] Connection header received
0620 18:37:00.507755 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:00.507761 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:00.507976 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:00.508113 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:00.509030 (+ 917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:00.509038 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:00.509049 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:37:00.509061 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:00.509105 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:37:00.509121 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:00.509258 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:00.509422 (+ 164us) server_negotiation.cc:299] Negotiation successful
0620 18:37:00.509589 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":300,"threads_started":1}
I20260620 18:37:00.511202 461 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:37:00.591380 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 339
I20260620 18:37:00.608070 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 32456
I20260620 18:37:00.609220 32517 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:37:00.710749 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 32456
2026-06-20T18:37:00Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2272 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980527431848-22676-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:37:00 dist-test-slave-5lgp krb5kdc[484](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:37:00 dist-test-slave-5lgp krb5kdc[484](info): set up 2 sockets
Jun 20 18:37:00 dist-test-slave-5lgp krb5kdc[484](info): commencing operation
krb5kdc: starting...
W20260620 18:37:01.818332 22676 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980527431848-22676-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980527431848-22676-0/krb5kdc/test-user.keytab.
Jun 20 18:37:01 dist-test-slave-5lgp krb5kdc[484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980621, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:37:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:37:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.37.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.37.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.37.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab.
Entry for principal kudu/127.22.37.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab.
I20260620 18:37:02.051167 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980527431848-22676-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980527431848-22676-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980527431848-22676-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.62:38229
--webserver_interface=127.22.37.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab
--principal=kudu/127.22.37.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:36537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.37.62:38229
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:37:02.367353 497 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:37:02.367746 497 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:37:02.367918 497 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:37:02.375895 497 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:37:02.376051 497 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:37:02.376143 497 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:37:02.376228 497 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:37:02.376314 497 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:37:02.386400 497 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:36537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980527431848-22676-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980527431848-22676-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.37.62:38229
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab
--principal=kudu/127.22.37.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.62:38229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.37.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980527431848-22676-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:37:02.387768 497 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:37:02.389112 497 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:37:02.400516 502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:02.400609 505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:02.400624 503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:37:02.401301 497 server_base.cc:1034] running on GCE node
I20260620 18:37:02.402132 497 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:37:02.403532 497 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:37:02.404765 497 hybrid_clock.cc:648] HybridClock initialized: now 1781980622404727 us; error 46 us; skew 500 ppm
I20260620 18:37:02.405108 497 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:37:02 dist-test-slave-5lgp krb5kdc[484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980622, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:37:02.408171 497 init.cc:377] Logged in from keytab as kudu/127.22.37.62@KRBTEST.COM (short username kudu)
I20260620 18:37:02.409629 497 webserver.cc:460] Webserver started at http://127.22.37.62:35095/ using document root <none> and password file <none>
I20260620 18:37:02.410338 497 fs_manager.cc:359] Metadata directory not provided
I20260620 18:37:02.410478 497 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:37:02.410771 497 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:37:02.413697 497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980527431848-22676-0/minicluster-data/master-0/data/instance:
uuid: "5db3c289e5cb44a89bc6de4ecd50c46a"
format_stamp: "Formatted at 2026-06-20 18:37:02 on dist-test-slave-5lgp"
server_key: "4fd2a02c4716b4e08e23a81304937238"
server_key_iv: "a3d31ba1919d795c035b297767e9f82d"
server_key_version: "encryptionkey@0"
I20260620 18:37:02.414507 497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980527431848-22676-0/minicluster-data/master-0/wal/instance:
uuid: "5db3c289e5cb44a89bc6de4ecd50c46a"
format_stamp: "Formatted at 2026-06-20 18:37:02 on dist-test-slave-5lgp"
server_key: "4fd2a02c4716b4e08e23a81304937238"
server_key_iv: "a3d31ba1919d795c035b297767e9f82d"
server_key_version: "encryptionkey@0"
I20260620 18:37:02.419306 497 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 18:37:02.422657 512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:37:02.424023 497 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:37:02.424224 497 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980527431848-22676-0/minicluster-data/master-0/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980527431848-22676-0/minicluster-data/master-0/wal
uuid: "5db3c289e5cb44a89bc6de4ecd50c46a"
format_stamp: "Formatted at 2026-06-20 18:37:02 on dist-test-slave-5lgp"
server_key: "4fd2a02c4716b4e08e23a81304937238"
server_key_iv: "a3d31ba1919d795c035b297767e9f82d"
server_key_version: "encryptionkey@0"
I20260620 18:37:02.424453 497 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980527431848-22676-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980527431848-22676-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980527431848-22676-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:37:02.455688 497 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:37:02.460157 497 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:37:02.460453 497 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:37:02.478215 497 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.62:38229
I20260620 18:37:02.478248 563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.62:38229 every 8 connection(s)
I20260620 18:37:02.479521 497 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
I20260620 18:37:02.481706 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 497
I20260620 18:37:02.481936 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980527431848-22676-0/minicluster-data/master-0/wal/instance
I20260620 18:37:02.482573 22676 external_mini_cluster.cc:1428] Setting key 65f88a066d3c9ecaa40982392eb95812
I20260620 18:37:02.483330 564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:37:02.497004 564 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5db3c289e5cb44a89bc6de4ecd50c46a: Bootstrap starting.
Jun 20 18:37:02 dist-test-slave-5lgp krb5kdc[484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980621, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM
I20260620 18:37:02.500669 564 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5db3c289e5cb44a89bc6de4ecd50c46a: Neither blocks nor log segments found. Creating new log.
I20260620 18:37:02.501760 564 log.cc:824] T 00000000000000000000000000000000 P 5db3c289e5cb44a89bc6de4ecd50c46a: Log is configured to *not* fsync() on all Append() calls
I20260620 18:37:02.505146 564 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5db3c289e5cb44a89bc6de4ecd50c46a: No bootstrap required, opened a new log
I20260620 18:37:02.507035 567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:02.484579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53544 (local address 127.22.37.62:38229)
0620 18:37:02.485343 (+ 764us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:02.485358 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:02.485426 (+ 68us) server_negotiation.cc:407] Connection header received
0620 18:37:02.487839 (+ 2413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:02.487868 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:02.488368 (+ 500us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:02.488848 (+ 480us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:02.490239 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:02.492138 (+ 1899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:02.493528 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:02.494652 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:02.494844 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:02.499322 (+ 4478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:02.499330 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:02.499344 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:37:02.499379 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:02.502607 (+ 3228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:02.503390 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:02.503402 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:02.503412 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:02.503617 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:02.503981 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:02.503986 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:02.503989 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:02.504667 (+ 678us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:02.504961 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:02.505233 (+ 272us) server_negotiation.cc:299] Negotiation successful
0620 18:37:02.505709 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":317,"threads_started":1}
I20260620 18:37:02.514142 564 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5db3c289e5cb44a89bc6de4ecd50c46a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5db3c289e5cb44a89bc6de4ecd50c46a" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 38229 } }
I20260620 18:37:02.514537 564 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5db3c289e5cb44a89bc6de4ecd50c46a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:37:02.514631 564 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5db3c289e5cb44a89bc6de4ecd50c46a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5db3c289e5cb44a89bc6de4ecd50c46a, State: Initialized, Role: FOLLOWER
I20260620 18:37:02.515110 564 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5db3c289e5cb44a89bc6de4ecd50c46a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5db3c289e5cb44a89bc6de4ecd50c46a" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 38229 } }
I20260620 18:37:02.515340 564 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5db3c289e5cb44a89bc6de4ecd50c46a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:37:02.515480 564 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5db3c289e5cb44a89bc6de4ecd50c46a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:37:02.515646 564 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5db3c289e5cb44a89bc6de4ecd50c46a [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:37:02.517967 564 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5db3c289e5cb44a89bc6de4ecd50c46a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5db3c289e5cb44a89bc6de4ecd50c46a" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 38229 } }
I20260620 18:37:02.518440 564 leader_election.cc:302] T 00000000000000000000000000000000 P 5db3c289e5cb44a89bc6de4ecd50c46a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5db3c289e5cb44a89bc6de4ecd50c46a; no voters:
I20260620 18:37:02.519055 564 leader_election.cc:288] T 00000000000000000000000000000000 P 5db3c289e5cb44a89bc6de4ecd50c46a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:37:02.519191 569 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5db3c289e5cb44a89bc6de4ecd50c46a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:37:02.519995 569 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5db3c289e5cb44a89bc6de4ecd50c46a [term 1 LEADER]: Becoming Leader. State: Replica: 5db3c289e5cb44a89bc6de4ecd50c46a, State: Running, Role: LEADER
I20260620 18:37:02.520483 569 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5db3c289e5cb44a89bc6de4ecd50c46a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5db3c289e5cb44a89bc6de4ecd50c46a" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 38229 } }
I20260620 18:37:02.521390 564 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5db3c289e5cb44a89bc6de4ecd50c46a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:37:02.523465 570 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5db3c289e5cb44a89bc6de4ecd50c46a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5db3c289e5cb44a89bc6de4ecd50c46a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5db3c289e5cb44a89bc6de4ecd50c46a" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 38229 } } }
I20260620 18:37:02.523831 570 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5db3c289e5cb44a89bc6de4ecd50c46a [sys.catalog]: This master's current role is: LEADER
I20260620 18:37:02.524366 571 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5db3c289e5cb44a89bc6de4ecd50c46a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5db3c289e5cb44a89bc6de4ecd50c46a. Latest consensus state: current_term: 1 leader_uuid: "5db3c289e5cb44a89bc6de4ecd50c46a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5db3c289e5cb44a89bc6de4ecd50c46a" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 38229 } } }
I20260620 18:37:02.524731 571 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5db3c289e5cb44a89bc6de4ecd50c46a [sys.catalog]: This master's current role is: LEADER
I20260620 18:37:02.530223 575 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:37:02.536103 575 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:37:02.545034 575 catalog_manager.cc:1347] Generated new cluster ID: 080a6767a22f472fabf4b70cafaf0768
I20260620 18:37:02.545166 575 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:37:02.555769 575 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:37:02.557072 575 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:37:02.564724 575 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5db3c289e5cb44a89bc6de4ecd50c46a: Generated new TSK 0
I20260620 18:37:02.565580 575 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.37.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.37.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.37.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.1.keytab.
Entry for principal kudu/127.22.37.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.1.keytab.
I20260620 18:37:02.631067 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980527431848-22676-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980527431848-22676-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980527431848-22676-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980527431848-22676-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.37.1:0
--local_ip_for_outbound_sockets=127.22.37.1
--webserver_interface=127.22.37.1
--webserver_port=0
--tserver_master_addrs=127.22.37.62:38229
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.1.keytab
--principal=kudu/127.22.37.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:36537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:37:02.984490 591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:37:02.984861 591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:37:02.985056 591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:37:02.992651 591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:37:02.992851 591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:37:02.993048 591 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.37.1
I20260620 18:37:03.003870 591 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:36537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980527431848-22676-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980527431848-22676-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.1.keytab
--principal=kudu/127.22.37.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.37.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980527431848-22676-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.37.1
--webserver_port=0
--tserver_master_addrs=127.22.37.62:38229
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.37.1
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980527431848-22676-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:37:03.005304 591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:37:03.006716 591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:37:03.020730 597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:03.020752 596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:03.021502 599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:37:03.021593 591 server_base.cc:1034] running on GCE node
I20260620 18:37:03.022239 591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:37:03.023208 591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:37:03.024451 591 hybrid_clock.cc:648] HybridClock initialized: now 1781980623024404 us; error 60 us; skew 500 ppm
I20260620 18:37:03.024782 591 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:37:03 dist-test-slave-5lgp krb5kdc[484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980623, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:37:03.028280 591 init.cc:377] Logged in from keytab as kudu/127.22.37.1@KRBTEST.COM (short username kudu)
I20260620 18:37:03.029785 591 webserver.cc:460] Webserver started at http://127.22.37.1:37455/ using document root <none> and password file <none>
I20260620 18:37:03.030463 591 fs_manager.cc:359] Metadata directory not provided
I20260620 18:37:03.030614 591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:37:03.030912 591 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:37:03.034829 591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980527431848-22676-0/minicluster-data/ts-0/data/instance:
uuid: "e9dcf96f27444faa9d1238d1b948ded0"
format_stamp: "Formatted at 2026-06-20 18:37:03 on dist-test-slave-5lgp"
server_key: "573b99c8f7136df4eb7efed41101fc51"
server_key_iv: "173b08fe1e1b30f106b97b384208170c"
server_key_version: "encryptionkey@0"
I20260620 18:37:03.035627 591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980527431848-22676-0/minicluster-data/ts-0/wal/instance:
uuid: "e9dcf96f27444faa9d1238d1b948ded0"
format_stamp: "Formatted at 2026-06-20 18:37:03 on dist-test-slave-5lgp"
server_key: "573b99c8f7136df4eb7efed41101fc51"
server_key_iv: "173b08fe1e1b30f106b97b384208170c"
server_key_version: "encryptionkey@0"
I20260620 18:37:03.040693 591 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260620 18:37:03.044059 606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:37:03.045316 591 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:37:03.045528 591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980527431848-22676-0/minicluster-data/ts-0/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980527431848-22676-0/minicluster-data/ts-0/wal
uuid: "e9dcf96f27444faa9d1238d1b948ded0"
format_stamp: "Formatted at 2026-06-20 18:37:03 on dist-test-slave-5lgp"
server_key: "573b99c8f7136df4eb7efed41101fc51"
server_key_iv: "173b08fe1e1b30f106b97b384208170c"
server_key_version: "encryptionkey@0"
I20260620 18:37:03.045732 591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980527431848-22676-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980527431848-22676-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980527431848-22676-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:37:03.056563 591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:37:03.060449 591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:37:03.060757 591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:37:03.061694 591 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:37:03.062948 591 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:37:03.063051 591 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:37:03.063239 591 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:37:03.063324 591 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:37:03.095906 591 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.1:34907
I20260620 18:37:03.095952 718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.1:34907 every 8 connection(s)
I20260620 18:37:03.097249 591 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980527431848-22676-0/minicluster-data/ts-0/data/info.pb
I20260620 18:37:03.106861 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 591
I20260620 18:37:03.107084 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980527431848-22676-0/minicluster-data/ts-0/wal/instance
I20260620 18:37:03.107764 22676 external_mini_cluster.cc:1428] Setting key 7d11b3e2dd3947dec154d4fe3b2bd67b
Jun 20 18:37:03 dist-test-slave-5lgp krb5kdc[484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980623, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.1@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM
I20260620 18:37:03.118521 721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:03.099869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.1:33563 (local address 127.22.37.62:38229)
0620 18:37:03.100324 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:03.100328 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:03.101202 (+ 874us) server_negotiation.cc:407] Connection header received
0620 18:37:03.103088 (+ 1886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:03.103093 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:03.103198 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:03.103314 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:03.104805 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:03.105612 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:03.107306 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:03.107740 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:03.107883 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:03.111586 (+ 3703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:03.111594 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:03.111602 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:37:03.111643 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:03.115216 (+ 3573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:03.116090 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:03.116097 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:03.116103 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:03.116244 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:03.116683 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:03.116690 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:03.116696 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:03.117080 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:03.117235 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:03.118096 (+ 861us) server_negotiation.cc:299] Negotiation successful
0620 18:37:03.118263 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":247,"threads_started":1}
I20260620 18:37:03.120131 722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:03.100344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:38229 (local address 127.22.37.1:33563)
0620 18:37:03.101035 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:03.101072 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 18:37:03.102805 (+ 1733us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:37:03.103559 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:37:03.103568 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:37:03.104079 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:37:03.104637 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:03.104654 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:03.105763 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:03.105768 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:03.107113 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:03.107151 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:03.107971 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:03.107980 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:03.108222 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:03.108908 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:37:03.108930 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:37:03.111356 (+ 2426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:37:03.115416 (+ 4060us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:37:03.115426 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:37:03.115445 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:37:03.115958 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:37:03.116351 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:37:03.116354 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:37:03.116356 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:37:03.116560 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:37:03.117199 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:37:03.117204 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:37:03.117682 (+ 478us) client_negotiation.cc:769] Sending connection context
0620 18:37:03.118017 (+ 335us) client_negotiation.cc:240] Negotiation successful
0620 18:37:03.118647 (+ 630us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":273,"threads_started":1}
I20260620 18:37:03.121836 719 heartbeater.cc:347] Connected to a master server at 127.22.37.62:38229
I20260620 18:37:03.122179 719 heartbeater.cc:464] Registering TS with master...
I20260620 18:37:03.122901 719 heartbeater.cc:511] Master 127.22.37.62:38229 requested a full tablet report, sending...
I20260620 18:37:03.124854 529 ts_manager.cc:194] Registered new tserver with Master: e9dcf96f27444faa9d1238d1b948ded0 (127.22.37.1:34907)
I20260620 18:37:03.126418 529 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.37.1@KRBTEST.COM'} at 127.22.37.1:33563
WARNING: no policy specified for kudu/127.22.37.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.37.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.37.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.2.keytab.
Entry for principal kudu/127.22.37.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.2.keytab.
I20260620 18:37:03.164321 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980527431848-22676-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980527431848-22676-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980527431848-22676-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980527431848-22676-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.37.2:0
--local_ip_for_outbound_sockets=127.22.37.2
--webserver_interface=127.22.37.2
--webserver_port=0
--tserver_master_addrs=127.22.37.62:38229
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.2.keytab
--principal=kudu/127.22.37.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:36537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:37:03.488392 725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:37:03.488799 725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:37:03.488938 725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:37:03.500589 725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:37:03.500725 725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:37:03.500902 725 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.37.2
I20260620 18:37:03.511521 725 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:36537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980527431848-22676-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980527431848-22676-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.2.keytab
--principal=kudu/127.22.37.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.37.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980527431848-22676-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.37.2
--webserver_port=0
--tserver_master_addrs=127.22.37.62:38229
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.37.2
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980527431848-22676-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:37:03.512755 725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:37:03.514072 725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:37:03.527082 730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:03.527204 731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:03.527099 733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:37:03.528630 725 server_base.cc:1034] running on GCE node
I20260620 18:37:03.529292 725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:37:03.530272 725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:37:03.531482 725 hybrid_clock.cc:648] HybridClock initialized: now 1781980623531432 us; error 39 us; skew 500 ppm
I20260620 18:37:03.531785 725 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:37:03 dist-test-slave-5lgp krb5kdc[484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980623, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:37:03.535074 725 init.cc:377] Logged in from keytab as kudu/127.22.37.2@KRBTEST.COM (short username kudu)
I20260620 18:37:03.536543 725 webserver.cc:460] Webserver started at http://127.22.37.2:35159/ using document root <none> and password file <none>
I20260620 18:37:03.537237 725 fs_manager.cc:359] Metadata directory not provided
I20260620 18:37:03.537357 725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:37:03.537725 725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:37:03.540565 725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980527431848-22676-0/minicluster-data/ts-1/data/instance:
uuid: "0f774ac0ddc74c1ebc289f5bbaf301c2"
format_stamp: "Formatted at 2026-06-20 18:37:03 on dist-test-slave-5lgp"
server_key: "ec921fbbabdc35ff7aaf44760f324e90"
server_key_iv: "35a7c6718bc3b607fc234563eecd2d3d"
server_key_version: "encryptionkey@0"
I20260620 18:37:03.541311 725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980527431848-22676-0/minicluster-data/ts-1/wal/instance:
uuid: "0f774ac0ddc74c1ebc289f5bbaf301c2"
format_stamp: "Formatted at 2026-06-20 18:37:03 on dist-test-slave-5lgp"
server_key: "ec921fbbabdc35ff7aaf44760f324e90"
server_key_iv: "35a7c6718bc3b607fc234563eecd2d3d"
server_key_version: "encryptionkey@0"
I20260620 18:37:03.546303 725 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 18:37:03.549683 740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:37:03.550947 725 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 18:37:03.551108 725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980527431848-22676-0/minicluster-data/ts-1/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980527431848-22676-0/minicluster-data/ts-1/wal
uuid: "0f774ac0ddc74c1ebc289f5bbaf301c2"
format_stamp: "Formatted at 2026-06-20 18:37:03 on dist-test-slave-5lgp"
server_key: "ec921fbbabdc35ff7aaf44760f324e90"
server_key_iv: "35a7c6718bc3b607fc234563eecd2d3d"
server_key_version: "encryptionkey@0"
I20260620 18:37:03.551314 725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980527431848-22676-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980527431848-22676-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980527431848-22676-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:37:03.569245 725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:37:03.573486 725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:37:03.573770 725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:37:03.574692 725 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:37:03.576103 725 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:37:03.576205 725 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:37:03.576355 725 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:37:03.576435 725 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:37:03.607836 725 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.2:37023
I20260620 18:37:03.607878 852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.2:37023 every 8 connection(s)
I20260620 18:37:03.609126 725 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980527431848-22676-0/minicluster-data/ts-1/data/info.pb
I20260620 18:37:03.615382 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 725
I20260620 18:37:03.615595 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980527431848-22676-0/minicluster-data/ts-1/wal/instance
I20260620 18:37:03.616092 22676 external_mini_cluster.cc:1428] Setting key c6b8359181f61fd550856e5c251864ba
Jun 20 18:37:03 dist-test-slave-5lgp krb5kdc[484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980623, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.2@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM
I20260620 18:37:03.630748 721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:03.611652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.2:54899 (local address 127.22.37.62:38229)
0620 18:37:03.611853 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:03.611859 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:03.613077 (+ 1218us) server_negotiation.cc:407] Connection header received
0620 18:37:03.614829 (+ 1752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:03.614834 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:03.614898 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:03.614999 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:03.616935 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:03.617954 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:03.619411 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:03.619823 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:03.619972 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:03.623992 (+ 4020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:03.623998 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:03.624002 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:37:03.624030 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:03.627400 (+ 3370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:03.628202 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:03.628211 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:03.628217 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:03.628377 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:03.628806 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:03.628814 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:03.628820 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:03.629186 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:03.629329 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:03.630035 (+ 706us) server_negotiation.cc:299] Negotiation successful
0620 18:37:03.630297 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 18:37:03.631434 855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:03.612218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:38229 (local address 127.22.37.2:54899)
0620 18:37:03.612903 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:03.612941 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 18:37:03.614582 (+ 1641us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:37:03.615290 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:37:03.615300 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:37:03.616008 (+ 708us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:37:03.616766 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:03.616786 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:03.618098 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:03.618103 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:03.619268 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:03.619279 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:03.620074 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:03.620082 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:03.620297 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:03.620963 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:37:03.620989 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:37:03.623773 (+ 2784us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:37:03.627521 (+ 3748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:37:03.627527 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:37:03.627581 (+ 54us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:37:03.628097 (+ 516us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:37:03.628482 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:37:03.628485 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:37:03.628488 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:37:03.628682 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:37:03.629324 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:37:03.629332 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:37:03.629677 (+ 345us) client_negotiation.cc:769] Sending connection context
0620 18:37:03.629901 (+ 224us) client_negotiation.cc:240] Negotiation successful
0620 18:37:03.630290 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":273,"threads_started":1}
I20260620 18:37:03.632912 853 heartbeater.cc:347] Connected to a master server at 127.22.37.62:38229
I20260620 18:37:03.633230 853 heartbeater.cc:464] Registering TS with master...
I20260620 18:37:03.634181 853 heartbeater.cc:511] Master 127.22.37.62:38229 requested a full tablet report, sending...
I20260620 18:37:03.635763 529 ts_manager.cc:194] Registered new tserver with Master: 0f774ac0ddc74c1ebc289f5bbaf301c2 (127.22.37.2:37023)
I20260620 18:37:03.636725 529 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.37.2@KRBTEST.COM'} at 127.22.37.2:54899
WARNING: no policy specified for kudu/127.22.37.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.37.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.37.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.3.keytab.
Entry for principal kudu/127.22.37.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.3.keytab.
I20260620 18:37:03.668680 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980527431848-22676-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980527431848-22676-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980527431848-22676-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980527431848-22676-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.37.3:0
--local_ip_for_outbound_sockets=127.22.37.3
--webserver_interface=127.22.37.3
--webserver_port=0
--tserver_master_addrs=127.22.37.62:38229
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.3.keytab
--principal=kudu/127.22.37.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:36537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:37:03.992662 858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:37:03.993014 858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:37:03.993188 858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:37:04.000679 858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:37:04.000850 858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:37:04.001031 858 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.37.3
I20260620 18:37:04.011099 858 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:36537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980527431848-22676-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980527431848-22676-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.3.keytab
--principal=kudu/127.22.37.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.37.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980527431848-22676-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.37.3
--webserver_port=0
--tserver_master_addrs=127.22.37.62:38229
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.37.3
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980527431848-22676-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:37:04.012547 858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:37:04.013953 858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:37:04.027515 866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:04.027558 863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:04.027527 864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:37:04.028528 858 server_base.cc:1034] running on GCE node
I20260620 18:37:04.029256 858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:37:04.030285 858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:37:04.031544 858 hybrid_clock.cc:648] HybridClock initialized: now 1781980624031489 us; error 54 us; skew 500 ppm
I20260620 18:37:04.031881 858 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:37:04 dist-test-slave-5lgp krb5kdc[484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980624, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:37:04.035475 858 init.cc:377] Logged in from keytab as kudu/127.22.37.3@KRBTEST.COM (short username kudu)
I20260620 18:37:04.037281 858 webserver.cc:460] Webserver started at http://127.22.37.3:43753/ using document root <none> and password file <none>
I20260620 18:37:04.038057 858 fs_manager.cc:359] Metadata directory not provided
I20260620 18:37:04.038203 858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:37:04.038514 858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:37:04.041620 858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980527431848-22676-0/minicluster-data/ts-2/data/instance:
uuid: "5a0ba97f24b846f8a82c35a312d56037"
format_stamp: "Formatted at 2026-06-20 18:37:04 on dist-test-slave-5lgp"
server_key: "c913265caf0ec6f383cb828e55204445"
server_key_iv: "4190022e9c8bf0636516bb68863f31c7"
server_key_version: "encryptionkey@0"
I20260620 18:37:04.042380 858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980527431848-22676-0/minicluster-data/ts-2/wal/instance:
uuid: "5a0ba97f24b846f8a82c35a312d56037"
format_stamp: "Formatted at 2026-06-20 18:37:04 on dist-test-slave-5lgp"
server_key: "c913265caf0ec6f383cb828e55204445"
server_key_iv: "4190022e9c8bf0636516bb68863f31c7"
server_key_version: "encryptionkey@0"
I20260620 18:37:04.047576 858 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:37:04.051337 873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:37:04.052743 858 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:37:04.052927 858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980527431848-22676-0/minicluster-data/ts-2/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980527431848-22676-0/minicluster-data/ts-2/wal
uuid: "5a0ba97f24b846f8a82c35a312d56037"
format_stamp: "Formatted at 2026-06-20 18:37:04 on dist-test-slave-5lgp"
server_key: "c913265caf0ec6f383cb828e55204445"
server_key_iv: "4190022e9c8bf0636516bb68863f31c7"
server_key_version: "encryptionkey@0"
I20260620 18:37:04.053129 858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980527431848-22676-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980527431848-22676-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980527431848-22676-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:37:04.083604 858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:37:04.087800 858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:37:04.088136 858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:37:04.089131 858 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:37:04.090598 858 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:37:04.090701 858 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:37:04.090855 858 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:37:04.090936 858 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:37:04.124583 858 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.3:46807
I20260620 18:37:04.124629 985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.3:46807 every 8 connection(s)
I20260620 18:37:04.126014 858 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980527431848-22676-0/minicluster-data/ts-2/data/info.pb
I20260620 18:37:04.129565 719 heartbeater.cc:503] Master 127.22.37.62:38229 was elected leader, sending a full tablet report...
I20260620 18:37:04.130715 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 858
I20260620 18:37:04.130913 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980527431848-22676-0/minicluster-data/ts-2/wal/instance
I20260620 18:37:04.131461 22676 external_mini_cluster.cc:1428] Setting key e3390c768524ecd9a9e1a8a47f0a6e6f
Jun 20 18:37:04 dist-test-slave-5lgp krb5kdc[484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980624, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.3@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM
I20260620 18:37:04.147095 721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:04.128784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.3:40233 (local address 127.22.37.62:38229)
0620 18:37:04.128980 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:04.128985 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:04.130678 (+ 1693us) server_negotiation.cc:407] Connection header received
0620 18:37:04.132978 (+ 2300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:04.132983 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:04.133106 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:04.133224 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:04.134807 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:04.135551 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:04.137211 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:04.137696 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:04.137829 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:04.141268 (+ 3439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:04.141279 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:04.141282 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:37:04.141311 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:04.144006 (+ 2695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:04.144742 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:04.144754 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:04.144757 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:04.144879 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:04.145315 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:04.145321 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:04.145325 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:04.145676 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:04.145827 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:04.146543 (+ 716us) server_negotiation.cc:299] Negotiation successful
0620 18:37:04.146755 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 18:37:04.148082 988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:04.129486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:38229 (local address 127.22.37.3:40233)
0620 18:37:04.130522 (+ 1036us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:04.130571 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 18:37:04.132698 (+ 2127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:37:04.133531 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:37:04.133540 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:37:04.134067 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:37:04.134646 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:04.134664 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:04.135703 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:04.135709 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:04.137074 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:04.137086 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:04.137921 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:04.137929 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:04.138175 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:04.138861 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:37:04.138883 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:37:04.141061 (+ 2178us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:37:04.144162 (+ 3101us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:37:04.144171 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:37:04.144189 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:37:04.144608 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:37:04.145004 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:37:04.145009 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:37:04.145014 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:37:04.145203 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:37:04.145804 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:37:04.145812 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:37:04.146165 (+ 353us) client_negotiation.cc:769] Sending connection context
0620 18:37:04.146429 (+ 264us) client_negotiation.cc:240] Negotiation successful
0620 18:37:04.146964 (+ 535us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":721,"thread_start_us":506,"threads_started":1}
I20260620 18:37:04.149623 986 heartbeater.cc:347] Connected to a master server at 127.22.37.62:38229
I20260620 18:37:04.149983 986 heartbeater.cc:464] Registering TS with master...
I20260620 18:37:04.150770 986 heartbeater.cc:511] Master 127.22.37.62:38229 requested a full tablet report, sending...
I20260620 18:37:04.152240 529 ts_manager.cc:194] Registered new tserver with Master: 5a0ba97f24b846f8a82c35a312d56037 (127.22.37.3:46807)
I20260620 18:37:04.153010 529 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.37.3@KRBTEST.COM'} at 127.22.37.3:40233
I20260620 18:37:04.159111 22676 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:37:04.175571 721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:04.163492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53548 (local address 127.22.37.62:38229)
0620 18:37:04.163729 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:04.163736 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:04.164007 (+ 271us) server_negotiation.cc:407] Connection header received
0620 18:37:04.164141 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:04.164144 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:04.164201 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:04.164279 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:04.165688 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:04.166427 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:04.167567 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:04.167962 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:04.168052 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:04.170411 (+ 2359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:04.170416 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:04.170418 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:37:04.170448 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:04.173010 (+ 2562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:04.173648 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:04.173652 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:04.173654 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:04.173770 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:04.174152 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:04.174156 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:04.174158 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:04.174544 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:04.174747 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:04.175035 (+ 288us) server_negotiation.cc:299] Negotiation successful
0620 18:37:04.175196 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 18:37:04.180004 529 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53548:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:37:04.186694 529 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:37:04.224555 1000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:04.198200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.2:37023 (local address 127.0.0.1:50146)
0620 18:37:04.214000 (+ 15800us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:04.214062 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 18:37:04.214308 (+ 246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:37:04.216023 (+ 1715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:37:04.216038 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:37:04.216088 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:37:04.216931 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:04.216944 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:04.218830 (+ 1886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:04.218836 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:04.221770 (+ 2934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:04.221785 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:04.223417 (+ 1632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:04.223423 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:04.223621 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:04.223685 (+ 64us) client_negotiation.cc:769] Sending connection context
0620 18:37:04.223819 (+ 134us) client_negotiation.cc:240] Negotiation successful
0620 18:37:04.224007 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15464,"mutex_wait_us":6,"spinlock_wait_cycles":16256,"thread_start_us":18087,"threads_started":1}
I20260620 18:37:04.224646 1003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:04.199345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56680 (local address 127.22.37.3:46807)
0620 18:37:04.207261 (+ 7916us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:04.207274 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:04.216395 (+ 9121us) server_negotiation.cc:407] Connection header received
0620 18:37:04.216705 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:04.216713 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:04.216976 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:04.217199 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:37:04.217897 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:04.219531 (+ 1634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:04.222159 (+ 2628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:04.223569 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:04.223696 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:04.223912 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:04.224121 (+ 209us) server_negotiation.cc:299] Negotiation successful
0620 18:37:04.224287 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7703,"thread_start_us":7558,"threads_started":1}
I20260620 18:37:04.225232 999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:04.198326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50146 (local address 127.22.37.2:37023)
0620 18:37:04.213106 (+ 14780us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:04.213118 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:04.214575 (+ 1457us) server_negotiation.cc:407] Connection header received
0620 18:37:04.214763 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:04.214774 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:04.215102 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:04.215332 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:37:04.217114 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:04.218690 (+ 1576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:04.221952 (+ 3262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:04.223231 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:04.223374 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:04.223579 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:04.224861 (+ 1282us) server_negotiation.cc:299] Negotiation successful
0620 18:37:04.225003 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14562,"thread_start_us":14790,"threads_started":1}
I20260620 18:37:04.224581 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:04.198314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.3:46807 (local address 127.0.0.1:56680)
0620 18:37:04.216250 (+ 17936us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:04.216272 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 18:37:04.216509 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:37:04.217177 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:37:04.217181 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:37:04.217204 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:37:04.217646 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:04.217662 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:04.219683 (+ 2021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:04.219688 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:04.222010 (+ 2322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:04.222022 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:04.223781 (+ 1759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:04.223790 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:04.223939 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:04.223968 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 18:37:04.224040 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 18:37:04.224171 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17798,"mutex_wait_us":78,"thread_start_us":15180,"threads_started":1}
I20260620 18:37:04.225250 998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:04.198710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.1:34907 (local address 127.0.0.1:44758)
0620 18:37:04.216562 (+ 17852us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:04.216580 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:37:04.216687 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:37:04.217547 (+ 860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:37:04.217552 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:37:04.217579 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:37:04.218223 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:04.218236 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:04.220083 (+ 1847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:04.220090 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:04.222345 (+ 2255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:04.222358 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:04.224481 (+ 2123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:04.224491 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:04.224679 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:04.224711 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 18:37:04.224789 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 18:37:04.224915 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17710,"mutex_wait_us":144,"thread_start_us":17270,"threads_started":1}
I20260620 18:37:04.225885 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:04.199311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44758 (local address 127.22.37.1:34907)
0620 18:37:04.215810 (+ 16499us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:04.215822 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:04.216680 (+ 858us) server_negotiation.cc:407] Connection header received
0620 18:37:04.216828 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:04.216833 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:04.216996 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:04.217210 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:37:04.218602 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:04.219953 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:04.222515 (+ 2562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:04.223970 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:04.224122 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:04.224320 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:04.225415 (+ 1095us) server_negotiation.cc:299] Negotiation successful
0620 18:37:04.225583 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16299,"thread_start_us":16159,"threads_started":1}
I20260620 18:37:04.229930 921 tablet_service.cc:1467] Processing CreateTablet for tablet db6371c1bbdd47769b14e049660dd982 (DEFAULT_TABLE table=test-table [id=59772fadbbf7453cb6890fc6e85ebf92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:37:04.229930 654 tablet_service.cc:1467] Processing CreateTablet for tablet db6371c1bbdd47769b14e049660dd982 (DEFAULT_TABLE table=test-table [id=59772fadbbf7453cb6890fc6e85ebf92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:37:04.231434 654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db6371c1bbdd47769b14e049660dd982. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:37:04.231642 921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db6371c1bbdd47769b14e049660dd982. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:37:04.231791 788 tablet_service.cc:1467] Processing CreateTablet for tablet db6371c1bbdd47769b14e049660dd982 (DEFAULT_TABLE table=test-table [id=59772fadbbf7453cb6890fc6e85ebf92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:37:04.232802 788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db6371c1bbdd47769b14e049660dd982. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:37:04.244751 1004 tablet_bootstrap.cc:492] T db6371c1bbdd47769b14e049660dd982 P 0f774ac0ddc74c1ebc289f5bbaf301c2: Bootstrap starting.
I20260620 18:37:04.246340 1006 tablet_bootstrap.cc:492] T db6371c1bbdd47769b14e049660dd982 P 5a0ba97f24b846f8a82c35a312d56037: Bootstrap starting.
I20260620 18:37:04.248617 1004 tablet_bootstrap.cc:654] T db6371c1bbdd47769b14e049660dd982 P 0f774ac0ddc74c1ebc289f5bbaf301c2: Neither blocks nor log segments found. Creating new log.
I20260620 18:37:04.250006 1006 tablet_bootstrap.cc:654] T db6371c1bbdd47769b14e049660dd982 P 5a0ba97f24b846f8a82c35a312d56037: Neither blocks nor log segments found. Creating new log.
I20260620 18:37:04.249987 1004 log.cc:824] T db6371c1bbdd47769b14e049660dd982 P 0f774ac0ddc74c1ebc289f5bbaf301c2: Log is configured to *not* fsync() on all Append() calls
I20260620 18:37:04.250311 1005 tablet_bootstrap.cc:492] T db6371c1bbdd47769b14e049660dd982 P e9dcf96f27444faa9d1238d1b948ded0: Bootstrap starting.
I20260620 18:37:04.251541 1006 log.cc:824] T db6371c1bbdd47769b14e049660dd982 P 5a0ba97f24b846f8a82c35a312d56037: Log is configured to *not* fsync() on all Append() calls
I20260620 18:37:04.253695 1005 tablet_bootstrap.cc:654] T db6371c1bbdd47769b14e049660dd982 P e9dcf96f27444faa9d1238d1b948ded0: Neither blocks nor log segments found. Creating new log.
I20260620 18:37:04.254913 1005 log.cc:824] T db6371c1bbdd47769b14e049660dd982 P e9dcf96f27444faa9d1238d1b948ded0: Log is configured to *not* fsync() on all Append() calls
I20260620 18:37:04.255206 1006 tablet_bootstrap.cc:492] T db6371c1bbdd47769b14e049660dd982 P 5a0ba97f24b846f8a82c35a312d56037: No bootstrap required, opened a new log
I20260620 18:37:04.255508 1006 ts_tablet_manager.cc:1397] T db6371c1bbdd47769b14e049660dd982 P 5a0ba97f24b846f8a82c35a312d56037: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260620 18:37:04.255684 1004 tablet_bootstrap.cc:492] T db6371c1bbdd47769b14e049660dd982 P 0f774ac0ddc74c1ebc289f5bbaf301c2: No bootstrap required, opened a new log
I20260620 18:37:04.256029 1004 ts_tablet_manager.cc:1397] T db6371c1bbdd47769b14e049660dd982 P 0f774ac0ddc74c1ebc289f5bbaf301c2: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20260620 18:37:04.258077 1005 tablet_bootstrap.cc:492] T db6371c1bbdd47769b14e049660dd982 P e9dcf96f27444faa9d1238d1b948ded0: No bootstrap required, opened a new log
I20260620 18:37:04.258337 1005 ts_tablet_manager.cc:1397] T db6371c1bbdd47769b14e049660dd982 P e9dcf96f27444faa9d1238d1b948ded0: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260620 18:37:04.266130 1005 raft_consensus.cc:348] T db6371c1bbdd47769b14e049660dd982 P e9dcf96f27444faa9d1238d1b948ded0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f774ac0ddc74c1ebc289f5bbaf301c2" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 37023 } } peers { permanent_uuid: "5a0ba97f24b846f8a82c35a312d56037" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 46807 } } peers { permanent_uuid: "e9dcf96f27444faa9d1238d1b948ded0" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 34907 } }
I20260620 18:37:04.266561 1005 raft_consensus.cc:374] T db6371c1bbdd47769b14e049660dd982 P e9dcf96f27444faa9d1238d1b948ded0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:37:04.266662 1005 raft_consensus.cc:729] T db6371c1bbdd47769b14e049660dd982 P e9dcf96f27444faa9d1238d1b948ded0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9dcf96f27444faa9d1238d1b948ded0, State: Initialized, Role: FOLLOWER
I20260620 18:37:04.266777 1006 raft_consensus.cc:348] T db6371c1bbdd47769b14e049660dd982 P 5a0ba97f24b846f8a82c35a312d56037 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f774ac0ddc74c1ebc289f5bbaf301c2" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 37023 } } peers { permanent_uuid: "5a0ba97f24b846f8a82c35a312d56037" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 46807 } } peers { permanent_uuid: "e9dcf96f27444faa9d1238d1b948ded0" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 34907 } }
I20260620 18:37:04.267252 1006 raft_consensus.cc:374] T db6371c1bbdd47769b14e049660dd982 P 5a0ba97f24b846f8a82c35a312d56037 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:37:04.267349 1006 raft_consensus.cc:729] T db6371c1bbdd47769b14e049660dd982 P 5a0ba97f24b846f8a82c35a312d56037 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a0ba97f24b846f8a82c35a312d56037, State: Initialized, Role: FOLLOWER
I20260620 18:37:04.267206 1004 raft_consensus.cc:348] T db6371c1bbdd47769b14e049660dd982 P 0f774ac0ddc74c1ebc289f5bbaf301c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f774ac0ddc74c1ebc289f5bbaf301c2" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 37023 } } peers { permanent_uuid: "5a0ba97f24b846f8a82c35a312d56037" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 46807 } } peers { permanent_uuid: "e9dcf96f27444faa9d1238d1b948ded0" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 34907 } }
I20260620 18:37:04.267378 1005 consensus_queue.cc:260] T db6371c1bbdd47769b14e049660dd982 P e9dcf96f27444faa9d1238d1b948ded0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f774ac0ddc74c1ebc289f5bbaf301c2" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 37023 } } peers { permanent_uuid: "5a0ba97f24b846f8a82c35a312d56037" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 46807 } } peers { permanent_uuid: "e9dcf96f27444faa9d1238d1b948ded0" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 34907 } }
I20260620 18:37:04.267650 1004 raft_consensus.cc:374] T db6371c1bbdd47769b14e049660dd982 P 0f774ac0ddc74c1ebc289f5bbaf301c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:37:04.267764 1004 raft_consensus.cc:729] T db6371c1bbdd47769b14e049660dd982 P 0f774ac0ddc74c1ebc289f5bbaf301c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f774ac0ddc74c1ebc289f5bbaf301c2, State: Initialized, Role: FOLLOWER
I20260620 18:37:04.268334 1004 consensus_queue.cc:260] T db6371c1bbdd47769b14e049660dd982 P 0f774ac0ddc74c1ebc289f5bbaf301c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f774ac0ddc74c1ebc289f5bbaf301c2" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 37023 } } peers { permanent_uuid: "5a0ba97f24b846f8a82c35a312d56037" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 46807 } } peers { permanent_uuid: "e9dcf96f27444faa9d1238d1b948ded0" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 34907 } }
I20260620 18:37:04.269454 1005 ts_tablet_manager.cc:1428] T db6371c1bbdd47769b14e049660dd982 P e9dcf96f27444faa9d1238d1b948ded0: Time spent starting tablet: real 0.011s user 0.011s sys 0.001s
I20260620 18:37:04.269472 853 heartbeater.cc:503] Master 127.22.37.62:38229 was elected leader, sending a full tablet report...
I20260620 18:37:04.268217 1006 consensus_queue.cc:260] T db6371c1bbdd47769b14e049660dd982 P 5a0ba97f24b846f8a82c35a312d56037 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f774ac0ddc74c1ebc289f5bbaf301c2" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 37023 } } peers { permanent_uuid: "5a0ba97f24b846f8a82c35a312d56037" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 46807 } } peers { permanent_uuid: "e9dcf96f27444faa9d1238d1b948ded0" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 34907 } }
I20260620 18:37:04.270200 1004 ts_tablet_manager.cc:1428] T db6371c1bbdd47769b14e049660dd982 P 0f774ac0ddc74c1ebc289f5bbaf301c2: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260620 18:37:04.271142 986 heartbeater.cc:503] Master 127.22.37.62:38229 was elected leader, sending a full tablet report...
I20260620 18:37:04.271808 1006 ts_tablet_manager.cc:1428] T db6371c1bbdd47769b14e049660dd982 P 5a0ba97f24b846f8a82c35a312d56037: Time spent starting tablet: real 0.016s user 0.013s sys 0.000s
W20260620 18:37:04.349713 720 tablet.cc:2369] T db6371c1bbdd47769b14e049660dd982 P e9dcf96f27444faa9d1238d1b948ded0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:37:04.361133 854 tablet.cc:2369] T db6371c1bbdd47769b14e049660dd982 P 0f774ac0ddc74c1ebc289f5bbaf301c2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:37:04.377791 987 tablet.cc:2369] T db6371c1bbdd47769b14e049660dd982 P 5a0ba97f24b846f8a82c35a312d56037: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:37:04.478794 1012 raft_consensus.cc:482] T db6371c1bbdd47769b14e049660dd982 P 5a0ba97f24b846f8a82c35a312d56037 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:37:04.479146 1012 raft_consensus.cc:504] T db6371c1bbdd47769b14e049660dd982 P 5a0ba97f24b846f8a82c35a312d56037 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f774ac0ddc74c1ebc289f5bbaf301c2" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 37023 } } peers { permanent_uuid: "5a0ba97f24b846f8a82c35a312d56037" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 46807 } } peers { permanent_uuid: "e9dcf96f27444faa9d1238d1b948ded0" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 34907 } }
I20260620 18:37:04.480842 1012 leader_election.cc:288] T db6371c1bbdd47769b14e049660dd982 P 5a0ba97f24b846f8a82c35a312d56037 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f774ac0ddc74c1ebc289f5bbaf301c2 (127.22.37.2:37023), e9dcf96f27444faa9d1238d1b948ded0 (127.22.37.1:34907)
I20260620 18:37:04.488095 999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:04.481697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.3:51661 (local address 127.22.37.2:37023)
0620 18:37:04.481881 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:04.481887 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:04.482253 (+ 366us) server_negotiation.cc:407] Connection header received
0620 18:37:04.482508 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:04.482512 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:04.482585 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:04.482692 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:37:04.483225 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:04.484035 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:04.485867 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:04.487089 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:04.487252 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:04.487352 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:04.487705 (+ 353us) server_negotiation.cc:299] Negotiation successful
0620 18:37:04.487837 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 18:37:04.488137 988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:04.481113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.1:34907 (local address 127.22.37.3:46555)
0620 18:37:04.481502 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:04.481524 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 18:37:04.481659 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:37:04.482172 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:37:04.482178 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:37:04.482209 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:37:04.482403 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:04.482413 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:04.483622 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:04.483627 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:04.485757 (+ 2130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:04.485772 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:04.487443 (+ 1671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:04.487447 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:04.487573 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:04.487595 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 18:37:04.487708 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 18:37:04.487840 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"mutex_wait_us":33}
I20260620 18:37:04.488327 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:04.481636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.3:46555 (local address 127.22.37.1:34907)
0620 18:37:04.481847 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:04.481855 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:04.481876 (+ 21us) server_negotiation.cc:407] Connection header received
0620 18:37:04.481959 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:04.481965 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:04.482041 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:04.482146 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:37:04.482556 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:04.483485 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:04.485933 (+ 2448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:04.487141 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:04.487256 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:04.487354 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:04.487956 (+ 602us) server_negotiation.cc:299] Negotiation successful
0620 18:37:04.488101 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 18:37:04.488751 1013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:04.481226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.2:37023 (local address 127.22.37.3:51661)
0620 18:37:04.482164 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:04.482184 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:37:04.482352 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:37:04.482722 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:37:04.482727 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:37:04.482751 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:37:04.483065 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:04.483073 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:04.484194 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:04.484200 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:04.485712 (+ 1512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:04.485724 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:04.487317 (+ 1593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:04.487323 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:04.487518 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:04.487563 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 18:37:04.487654 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 18:37:04.487793 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":812,"mutex_wait_us":26,"thread_start_us":603,"threads_started":1}
I20260620 18:37:04.489472 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db6371c1bbdd47769b14e049660dd982" candidate_uuid: "5a0ba97f24b846f8a82c35a312d56037" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9dcf96f27444faa9d1238d1b948ded0" is_pre_election: true
I20260620 18:37:04.489883 808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db6371c1bbdd47769b14e049660dd982" candidate_uuid: "5a0ba97f24b846f8a82c35a312d56037" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f774ac0ddc74c1ebc289f5bbaf301c2" is_pre_election: true
I20260620 18:37:04.490087 674 raft_consensus.cc:2454] T db6371c1bbdd47769b14e049660dd982 P e9dcf96f27444faa9d1238d1b948ded0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a0ba97f24b846f8a82c35a312d56037 in term 0.
I20260620 18:37:04.490410 808 raft_consensus.cc:2454] T db6371c1bbdd47769b14e049660dd982 P 0f774ac0ddc74c1ebc289f5bbaf301c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a0ba97f24b846f8a82c35a312d56037 in term 0.
I20260620 18:37:04.490903 876 leader_election.cc:302] T db6371c1bbdd47769b14e049660dd982 P 5a0ba97f24b846f8a82c35a312d56037 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a0ba97f24b846f8a82c35a312d56037, e9dcf96f27444faa9d1238d1b948ded0; no voters:
I20260620 18:37:04.491333 1012 raft_consensus.cc:2781] T db6371c1bbdd47769b14e049660dd982 P 5a0ba97f24b846f8a82c35a312d56037 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:37:04.491489 1012 raft_consensus.cc:482] T db6371c1bbdd47769b14e049660dd982 P 5a0ba97f24b846f8a82c35a312d56037 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:37:04.491633 1012 raft_consensus.cc:3037] T db6371c1bbdd47769b14e049660dd982 P 5a0ba97f24b846f8a82c35a312d56037 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:37:04.493870 1012 raft_consensus.cc:504] T db6371c1bbdd47769b14e049660dd982 P 5a0ba97f24b846f8a82c35a312d56037 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f774ac0ddc74c1ebc289f5bbaf301c2" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 37023 } } peers { permanent_uuid: "5a0ba97f24b846f8a82c35a312d56037" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 46807 } } peers { permanent_uuid: "e9dcf96f27444faa9d1238d1b948ded0" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 34907 } }
I20260620 18:37:04.494514 1012 leader_election.cc:288] T db6371c1bbdd47769b14e049660dd982 P 5a0ba97f24b846f8a82c35a312d56037 [CANDIDATE]: Term 1 election: Requested vote from peers 0f774ac0ddc74c1ebc289f5bbaf301c2 (127.22.37.2:37023), e9dcf96f27444faa9d1238d1b948ded0 (127.22.37.1:34907)
I20260620 18:37:04.494974 808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db6371c1bbdd47769b14e049660dd982" candidate_uuid: "5a0ba97f24b846f8a82c35a312d56037" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f774ac0ddc74c1ebc289f5bbaf301c2"
I20260620 18:37:04.495047 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db6371c1bbdd47769b14e049660dd982" candidate_uuid: "5a0ba97f24b846f8a82c35a312d56037" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9dcf96f27444faa9d1238d1b948ded0"
I20260620 18:37:04.495188 808 raft_consensus.cc:3037] T db6371c1bbdd47769b14e049660dd982 P 0f774ac0ddc74c1ebc289f5bbaf301c2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:37:04.495301 674 raft_consensus.cc:3037] T db6371c1bbdd47769b14e049660dd982 P e9dcf96f27444faa9d1238d1b948ded0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:37:04.497442 808 raft_consensus.cc:2454] T db6371c1bbdd47769b14e049660dd982 P 0f774ac0ddc74c1ebc289f5bbaf301c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a0ba97f24b846f8a82c35a312d56037 in term 1.
I20260620 18:37:04.497543 674 raft_consensus.cc:2454] T db6371c1bbdd47769b14e049660dd982 P e9dcf96f27444faa9d1238d1b948ded0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a0ba97f24b846f8a82c35a312d56037 in term 1.
I20260620 18:37:04.497941 876 leader_election.cc:302] T db6371c1bbdd47769b14e049660dd982 P 5a0ba97f24b846f8a82c35a312d56037 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0f774ac0ddc74c1ebc289f5bbaf301c2, 5a0ba97f24b846f8a82c35a312d56037, e9dcf96f27444faa9d1238d1b948ded0; no voters:
I20260620 18:37:04.498278 1012 raft_consensus.cc:2781] T db6371c1bbdd47769b14e049660dd982 P 5a0ba97f24b846f8a82c35a312d56037 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:37:04.498737 1012 raft_consensus.cc:686] T db6371c1bbdd47769b14e049660dd982 P 5a0ba97f24b846f8a82c35a312d56037 [term 1 LEADER]: Becoming Leader. State: Replica: 5a0ba97f24b846f8a82c35a312d56037, State: Running, Role: LEADER
I20260620 18:37:04.499217 1012 consensus_queue.cc:237] T db6371c1bbdd47769b14e049660dd982 P 5a0ba97f24b846f8a82c35a312d56037 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f774ac0ddc74c1ebc289f5bbaf301c2" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 37023 } } peers { permanent_uuid: "5a0ba97f24b846f8a82c35a312d56037" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 46807 } } peers { permanent_uuid: "e9dcf96f27444faa9d1238d1b948ded0" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 34907 } }
I20260620 18:37:04.503675 527 catalog_manager.cc:5495] T db6371c1bbdd47769b14e049660dd982 P 5a0ba97f24b846f8a82c35a312d56037 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a0ba97f24b846f8a82c35a312d56037 (127.22.37.3). New cstate: current_term: 1 leader_uuid: "5a0ba97f24b846f8a82c35a312d56037" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f774ac0ddc74c1ebc289f5bbaf301c2" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 37023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a0ba97f24b846f8a82c35a312d56037" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 46807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9dcf96f27444faa9d1238d1b948ded0" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 34907 } health_report { overall_health: UNKNOWN } } }
I20260620 18:37:04.520753 1003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:04.515579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56696 (local address 127.22.37.3:46807)
0620 18:37:04.515775 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:04.515779 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:04.515873 (+ 94us) server_negotiation.cc:407] Connection header received
0620 18:37:04.516045 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:04.516049 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:04.516131 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:04.516267 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:37:04.516585 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:04.517325 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:04.518844 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:04.519257 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:04.519365 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:04.519788 (+ 423us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:37:04.520307 (+ 519us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:37:04.520423 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:04.520487 (+ 64us) server_negotiation.cc:299] Negotiation successful
0620 18:37:04.520555 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 18:37:04.529517 808 raft_consensus.cc:1261] T db6371c1bbdd47769b14e049660dd982 P 0f774ac0ddc74c1ebc289f5bbaf301c2 [term 1 FOLLOWER]: Refusing update from remote peer 5a0ba97f24b846f8a82c35a312d56037: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:37:04.529524 674 raft_consensus.cc:1261] T db6371c1bbdd47769b14e049660dd982 P e9dcf96f27444faa9d1238d1b948ded0 [term 1 FOLLOWER]: Refusing update from remote peer 5a0ba97f24b846f8a82c35a312d56037: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:37:04.530616 1014 consensus_queue.cc:1035] T db6371c1bbdd47769b14e049660dd982 P 5a0ba97f24b846f8a82c35a312d56037 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f774ac0ddc74c1ebc289f5bbaf301c2" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 37023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:37:04.530958 1012 consensus_queue.cc:1035] T db6371c1bbdd47769b14e049660dd982 P 5a0ba97f24b846f8a82c35a312d56037 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9dcf96f27444faa9d1238d1b948ded0" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 34907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:37:04.542801 1017 mvcc.cc:205] Tried to move back new op lower bound from 7298992638061461504 to 7298992637952262144. Current Snapshot: MvccSnapshot[applied={T|T < 7298992638061461504}]
I20260620 18:37:04.557029 527 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53548:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:37:04.557331 527 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53548:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:37:04.561092 527 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5db3c289e5cb44a89bc6de4ecd50c46a: Sending DeleteTablet for 3 replicas of tablet db6371c1bbdd47769b14e049660dd982
I20260620 18:37:04.562256 921 tablet_service.cc:1514] Processing DeleteTablet for tablet db6371c1bbdd47769b14e049660dd982 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:37:04 UTC) from {username='kudu', principal='kudu/127.22.37.62@KRBTEST.COM'} at 127.0.0.1:56680
I20260620 18:37:04.562410 654 tablet_service.cc:1514] Processing DeleteTablet for tablet db6371c1bbdd47769b14e049660dd982 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:37:04 UTC) from {username='kudu', principal='kudu/127.22.37.62@KRBTEST.COM'} at 127.0.0.1:44758
I20260620 18:37:04.562439 788 tablet_service.cc:1514] Processing DeleteTablet for tablet db6371c1bbdd47769b14e049660dd982 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:37:04 UTC) from {username='kudu', principal='kudu/127.22.37.62@KRBTEST.COM'} at 127.0.0.1:50146
I20260620 18:37:04.563355 1027 tablet_replica.cc:330] T db6371c1bbdd47769b14e049660dd982 P e9dcf96f27444faa9d1238d1b948ded0: stopping tablet replica
I20260620 18:37:04.563961 1027 raft_consensus.cc:2229] T db6371c1bbdd47769b14e049660dd982 P e9dcf96f27444faa9d1238d1b948ded0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:37:04.564544 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 591
I20260620 18:37:04.564754 1027 raft_consensus.cc:2258] T db6371c1bbdd47769b14e049660dd982 P e9dcf96f27444faa9d1238d1b948ded0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:37:04.567564 1027 ts_tablet_manager.cc:1905] T db6371c1bbdd47769b14e049660dd982 P e9dcf96f27444faa9d1238d1b948ded0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:37:04.574939 1027 ts_tablet_manager.cc:1918] T db6371c1bbdd47769b14e049660dd982 P e9dcf96f27444faa9d1238d1b948ded0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:37:04.575639 1028 tablet_replica.cc:330] T db6371c1bbdd47769b14e049660dd982 P 0f774ac0ddc74c1ebc289f5bbaf301c2: stopping tablet replica
I20260620 18:37:04.576211 1028 raft_consensus.cc:2229] T db6371c1bbdd47769b14e049660dd982 P 0f774ac0ddc74c1ebc289f5bbaf301c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:37:04.576814 1028 raft_consensus.cc:2258] T db6371c1bbdd47769b14e049660dd982 P 0f774ac0ddc74c1ebc289f5bbaf301c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:37:04.577855 1027 log.cc:1196] T db6371c1bbdd47769b14e049660dd982 P e9dcf96f27444faa9d1238d1b948ded0: Deleting WAL directory at /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980527431848-22676-0/minicluster-data/ts-0/wal/wals/db6371c1bbdd47769b14e049660dd982
I20260620 18:37:04.579406 1028 ts_tablet_manager.cc:1905] T db6371c1bbdd47769b14e049660dd982 P 0f774ac0ddc74c1ebc289f5bbaf301c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:37:04.586696 1026 tablet_replica.cc:330] T db6371c1bbdd47769b14e049660dd982 P 5a0ba97f24b846f8a82c35a312d56037: stopping tablet replica
I20260620 18:37:04.587025 1027 ts_tablet_manager.cc:1939] T db6371c1bbdd47769b14e049660dd982 P e9dcf96f27444faa9d1238d1b948ded0: Deleting consensus metadata
I20260620 18:37:04.587234 1026 raft_consensus.cc:2229] T db6371c1bbdd47769b14e049660dd982 P 5a0ba97f24b846f8a82c35a312d56037 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:37:04.587996 1026 raft_consensus.cc:2258] T db6371c1bbdd47769b14e049660dd982 P 5a0ba97f24b846f8a82c35a312d56037 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:37:04.589038 515 catalog_manager.cc:4841] TS e9dcf96f27444faa9d1238d1b948ded0 (127.22.37.1:34907): tablet db6371c1bbdd47769b14e049660dd982 (table test-table [id=59772fadbbf7453cb6890fc6e85ebf92]) successfully deleted
I20260620 18:37:04.590000 1028 ts_tablet_manager.cc:1918] T db6371c1bbdd47769b14e049660dd982 P 0f774ac0ddc74c1ebc289f5bbaf301c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:37:04.590178 1028 log.cc:1196] T db6371c1bbdd47769b14e049660dd982 P 0f774ac0ddc74c1ebc289f5bbaf301c2: Deleting WAL directory at /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980527431848-22676-0/minicluster-data/ts-1/wal/wals/db6371c1bbdd47769b14e049660dd982
I20260620 18:37:04.590306 1026 ts_tablet_manager.cc:1905] T db6371c1bbdd47769b14e049660dd982 P 5a0ba97f24b846f8a82c35a312d56037: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 20 18:37:04 dist-test-slave-5lgp krb5kdc[484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980621, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.37.1@KRBTEST.COM
I20260620 18:37:04.590699 1028 ts_tablet_manager.cc:1939] T db6371c1bbdd47769b14e049660dd982 P 0f774ac0ddc74c1ebc289f5bbaf301c2: Deleting consensus metadata
I20260620 18:37:04.592496 514 catalog_manager.cc:4841] TS 0f774ac0ddc74c1ebc289f5bbaf301c2 (127.22.37.2:37023): tablet db6371c1bbdd47769b14e049660dd982 (table test-table [id=59772fadbbf7453cb6890fc6e85ebf92]) successfully deleted
I20260620 18:37:04.597635 1026 ts_tablet_manager.cc:1918] T db6371c1bbdd47769b14e049660dd982 P 5a0ba97f24b846f8a82c35a312d56037: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:37:04.597817 1026 log.cc:1196] T db6371c1bbdd47769b14e049660dd982 P 5a0ba97f24b846f8a82c35a312d56037: Deleting WAL directory at /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980527431848-22676-0/minicluster-data/ts-2/wal/wals/db6371c1bbdd47769b14e049660dd982
I20260620 18:37:04.598398 1026 ts_tablet_manager.cc:1939] T db6371c1bbdd47769b14e049660dd982 P 5a0ba97f24b846f8a82c35a312d56037: Deleting consensus metadata
I20260620 18:37:04.599380 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:04.567806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44766 (local address 127.22.37.1:34907)
0620 18:37:04.568033 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:04.568042 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:04.577074 (+ 9032us) server_negotiation.cc:407] Connection header received
0620 18:37:04.577699 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:04.577708 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:04.577805 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:04.577886 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:04.580074 (+ 2188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:04.581562 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:04.584067 (+ 2505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:04.584945 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:04.585068 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:04.591938 (+ 6870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:04.591955 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:04.591972 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:37:04.592028 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:04.595869 (+ 3841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:04.596811 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:04.596822 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:04.596825 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:04.596974 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:04.597464 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:04.597473 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:04.597476 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:04.598063 (+ 587us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:04.598384 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:04.598765 (+ 381us) server_negotiation.cc:299] Negotiation successful
0620 18:37:04.598984 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 18:37:04.600250 714 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:37:04.600890 516 catalog_manager.cc:4841] TS 5a0ba97f24b846f8a82c35a312d56037 (127.22.37.3:46807): tablet db6371c1bbdd47769b14e049660dd982 (table test-table [id=59772fadbbf7453cb6890fc6e85ebf92]) successfully deleted
I20260620 18:37:04.713816 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 591
I20260620 18:37:04.732622 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 725
Jun 20 18:37:04 dist-test-slave-5lgp krb5kdc[484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980621, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.37.2@KRBTEST.COM
I20260620 18:37:04.749457 999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:04.733497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50152 (local address 127.22.37.2:37023)
0620 18:37:04.733683 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:04.733690 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:04.733712 (+ 22us) server_negotiation.cc:407] Connection header received
0620 18:37:04.733814 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:04.733819 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:04.733919 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:04.734022 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:04.735182 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:04.736326 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:04.738126 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:04.738727 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:04.738850 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:04.742878 (+ 4028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:04.742897 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:04.742918 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:37:04.742965 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:04.746645 (+ 3680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:04.747271 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:04.747281 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:04.747285 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:04.747430 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:04.747795 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:04.747800 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:04.747803 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:04.748300 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:04.748586 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:04.748830 (+ 244us) server_negotiation.cc:299] Negotiation successful
0620 18:37:04.749050 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 18:37:04.750234 848 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:37:04.839955 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 725
I20260620 18:37:04.857533 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 858
Jun 20 18:37:04 dist-test-slave-5lgp krb5kdc[484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980621, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.37.3@KRBTEST.COM
I20260620 18:37:04.873560 1003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:04.858409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56708 (local address 127.22.37.3:46807)
0620 18:37:04.858571 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:04.858578 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:04.858674 (+ 96us) server_negotiation.cc:407] Connection header received
0620 18:37:04.858874 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:04.858880 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:04.858957 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:04.859077 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:04.860319 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:04.861111 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:04.862531 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:04.863007 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:04.863164 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:04.867458 (+ 4294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:04.867478 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:04.867494 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:37:04.867535 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:04.870712 (+ 3177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:04.871363 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:04.871372 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:04.871376 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:04.871539 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:04.871925 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:04.871931 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:04.871934 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:04.872286 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:04.872672 (+ 386us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:04.872900 (+ 228us) server_negotiation.cc:299] Negotiation successful
0620 18:37:04.873107 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 18:37:04.874248 981 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:37:04.995872 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 858
I20260620 18:37:05.015621 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 497
I20260620 18:37:05.016714 559 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:37:05.128722 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 497
2026-06-20T18:37:05Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4422 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:37:05 dist-test-slave-5lgp krb5kdc[1036](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:37:05 dist-test-slave-5lgp krb5kdc[1036](info): set up 2 sockets
Jun 20 18:37:05 dist-test-slave-5lgp krb5kdc[1036](info): commencing operation
krb5kdc: starting...
W20260620 18:37:06.230232 22676 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.003s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/krb5kdc/test-user.keytab.
Jun 20 18:37:06 dist-test-slave-5lgp krb5kdc[1036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980626, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:37:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:37:06Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.22.37.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.22.37.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.22.37.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/krb5kdc/rangeradmin_127.22.37.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.22.37.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/krb5kdc/rangeradmin_127.22.37.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.22.37.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.22.37.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.22.37.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/krb5kdc/rangerlookup_127.22.37.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.22.37.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/krb5kdc/rangerlookup_127.22.37.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.22.37.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.22.37.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.37.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/krb5kdc/HTTP_127.22.37.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.22.37.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/krb5kdc/HTTP_127.22.37.20@KRBTEST.COM.keytab.
I20260620 18:37:06.552673 22676 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task1UXpXs/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/minicluster-data/postgres -l logfile start
2026-06-20 18:37:09.055 UTC [1066] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 18:37:09.055 UTC [1066] LOG: listening on IPv4 address "127.22.37.20", port 45507
2026-06-20 18:37:09.059 UTC [1066] LOG: listening on Unix socket "/tmp/.s.PGSQL.45507"
2026-06-20 18:37:09.082 UTC [1069] LOG: database system was shut down at 2026-06-20 18:37:07 UTC
2026-06-20 18:37:09.085 UTC [1066] LOG: database system is ready to accept connections
I20260620 18:37:10.090106 22676 mini_postgres.cc:96] Postgres bound to 45507
2026-06-20 18:37:10.107 UTC [1077] FATAL: database "slave" does not exist
127.22.37.20:45507 - accepting connections
I20260620 18:37:10.108065 22676 mini_ranger.cc:161] Starting Ranger...
I20260620 18:37:10.133564 22676 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 18:37:10.604712 22676 mini_ranger.cc:87] Created ranger Postgres database
I20260620 18:37:10.604859 22676 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/minicluster-data/ranger-admin
2026-06-20 18:37:10,893 [I] DB FLAVOR :POSTGRES
2026-06-20 18:37:10,894 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 18:37:10,894 [I] Checking connection..
2026-06-20 18:37:10,894 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1UXpXs/build/asan/bin/postgresql.jar:/tmp/dist-test-task1UXpXs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.37.20:45507/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 18:37:11,153 [I] Checking connection passed.
2026-06-20 18:37:11,153 [I] --------- Verifying version history table ---------
2026-06-20 18:37:11,153 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1UXpXs/build/asan/bin/postgresql.jar:/tmp/dist-test-task1UXpXs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.37.20:45507/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 18:37:11,418 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 18:37:11,419 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1UXpXs/build/asan/bin/postgresql.jar:/tmp/dist-test-task1UXpXs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.37.20:45507/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 18:37:11,680 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 18:37:11,680 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 18:37:11,681 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1UXpXs/build/asan/bin/postgresql.jar:/tmp/dist-test-task1UXpXs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.37.20:45507/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task1UXpXs/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 18:37:11.914 UTC [1141] WARNING: there is no transaction in progress
2026-06-20 18:37:11,920 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 18:37:11,921 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1UXpXs/build/asan/bin/postgresql.jar:/tmp/dist-test-task1UXpXs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.37.20:45507/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 18:37:12,179 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 18:37:12,179 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 18:37:12,179 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1UXpXs/build/asan/bin/postgresql.jar:/tmp/dist-test-task1UXpXs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.37.20:45507/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 18:37:12,428 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1UXpXs/build/asan/bin/postgresql.jar:/tmp/dist-test-task1UXpXs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.37.20:45507/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 18:37:12,683 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1UXpXs/build/asan/bin/postgresql.jar:/tmp/dist-test-task1UXpXs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.37.20:45507/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-5lgp.c.gcp-upstream.internal','N') ;"
2026-06-20 18:37:12,922 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1UXpXs/build/asan/bin/postgresql.jar:/tmp/dist-test-task1UXpXs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.37.20:45507/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 18:37:13,177 [I] Table x_portal_user does not exist in database ranger
2026-06-20 18:37:13,177 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1UXpXs/build/asan/bin/postgresql.jar:/tmp/dist-test-task1UXpXs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.37.20:45507/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 18:37:13,429 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 18:37:13,430 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1UXpXs/build/asan/bin/postgresql.jar:/tmp/dist-test-task1UXpXs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.37.20:45507/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 18:37:13,685 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1UXpXs/build/asan/bin/postgresql.jar:/tmp/dist-test-task1UXpXs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.37.20:45507/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 18:37:13,933 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 18:37:13,933 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1UXpXs/build/asan/bin/postgresql.jar:/tmp/dist-test-task1UXpXs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.37.20:45507/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task1UXpXs/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 18:37:14.871 UTC [1249] WARNING: there is no transaction in progress
2026-06-20 18:37:14.886 UTC [1249] WARNING: there is no transaction in progress
2026-06-20 18:37:14.902 UTC [1249] WARNING: there is no transaction in progress
2026-06-20 18:37:14.917 UTC [1249] WARNING: there is no transaction in progress
2026-06-20 18:37:14.933 UTC [1249] WARNING: there is no transaction in progress
2026-06-20 18:37:14.951 UTC [1249] WARNING: there is no transaction in progress
2026-06-20 18:37:15.006 UTC [1249] WARNING: there is no transaction in progress
2026-06-20 18:37:15.013 UTC [1249] WARNING: there is no transaction in progress
2026-06-20 18:37:15.027 UTC [1249] WARNING: there is no transaction in progress
2026-06-20 18:37:15.038 UTC [1249] WARNING: there is no transaction in progress
2026-06-20 18:37:15.048 UTC [1249] WARNING: there is no transaction in progress
2026-06-20 18:37:15.062 UTC [1249] WARNING: there is no transaction in progress
2026-06-20 18:37:15.071 UTC [1249] WARNING: there is no transaction in progress
2026-06-20 18:37:15.078 UTC [1249] WARNING: there is no transaction in progress
2026-06-20 18:37:15.085 UTC [1249] WARNING: there is no transaction in progress
2026-06-20 18:37:15.091 UTC [1249] WARNING: there is no transaction in progress
2026-06-20 18:37:15,691 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 18:37:15,692 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1UXpXs/build/asan/bin/postgresql.jar:/tmp/dist-test-task1UXpXs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.37.20:45507/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 18:37:15,947 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 18:37:15,947 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1UXpXs/build/asan/bin/postgresql.jar:/tmp/dist-test-task1UXpXs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.37.20:45507/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 18:37:16,203 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 18:37:16,204 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1UXpXs/build/asan/bin/postgresql.jar:/tmp/dist-test-task1UXpXs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.37.20:45507/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 18:37:16,464 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 18:37:16,464 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1UXpXs/build/asan/bin/postgresql.jar:/tmp/dist-test-task1UXpXs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.37.20:45507/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 18:37:16,717 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1UXpXs/build/asan/bin/postgresql.jar:/tmp/dist-test-task1UXpXs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.37.20:45507/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 18:37:16,954 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1UXpXs/build/asan/bin/postgresql.jar:/tmp/dist-test-task1UXpXs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.37.20:45507/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-5lgp.c.gcp-upstream.internal';"
2026-06-20 18:37:17,208 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 18:37:17,209 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1UXpXs/build/asan/bin/postgresql.jar:/tmp/dist-test-task1UXpXs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.37.20:45507/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 18:37:17,435 [I] DB_PATCHES have already been applied
I20260620 18:37:17.441814 22676 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/minicluster-data/ranger-admin:/tmp/dist-test-task1UXpXs/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task1UXpXs/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task1UXpXs/build/asan/bin/postgresql.jar:/tmp/dist-test-task1UXpXs/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 18:37:17.441931 22676 mini_ranger.cc:193] Using host: 127.22.37.20
I20260620 18:37:17.457161 22676 mini_ranger.cc:227] Ranger admin URL: http://127.22.37.20:53273
Jun 20, 2026 6:37:17 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 6:37:17 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 6:37:17 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 6:37:17 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 6:37:17 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 6:37:18 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 6:37:18 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 18:37:18.600941 22676 mini_ranger.cc:160] Time spent starting Ranger: real 8.493s user 0.004s sys 0.036s
Jun 20, 2026 6:37:37 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 6:37:37 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Jun 20, 2026 6:37:37 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 6:37:37 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 6:37:37 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 6:37:37 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 6:37:37 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 6:37:37 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 6:37:37 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 6:37:37 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 6:37:37 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 6:37:37 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 6:37:37 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 6:37:37 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 6:37:37 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 6:37:37 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 6:37:37 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 6:37:38 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 18:37:39.072212 22676 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.22.37.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.22.37.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.22.37.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/krb5kdc/rangerkms_127.22.37.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.22.37.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/krb5kdc/rangerkms_127.22.37.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.22.37.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.22.37.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.37.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/krb5kdc/HTTP_127.22.37.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.22.37.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/krb5kdc/HTTP_127.22.37.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 18:37:39 dist-test-slave-5lgp krb5kdc[1036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980659, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 18:37:39.250397 22676 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 18:37:39.271277 22676 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 18:37:40.110123 22676 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 18:37:40.110298 22676 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/minicluster-data/ranger-kms
I20260620 18:37:40.110431 22676 mini_ranger_kms.cc:227] Using postgres at 127.22.37.20:45507
2026-06-20 18:37:40,526 [I] DB FLAVOR :POSTGRES
2026-06-20 18:37:40,527 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 18:37:40,527 [I] Checking connection
2026-06-20 18:37:40,527 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1UXpXs/build/asan/bin/postgresql.jar:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.37.20:45507/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 18:37:40,764 [I] connection success
2026-06-20 18:37:40,765 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 18:37:40,765 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 18:37:40,765 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1UXpXs/build/asan/bin/postgresql.jar:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.37.20:45507/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 18:37:40,997 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 18:37:40,997 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 18:37:40,997 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 18:37:40,997 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1UXpXs/build/asan/bin/postgresql.jar:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.37.20:45507/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 18:37:41,256 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 18:37:41.690567 22676 mini_ranger_kms.cc:310] Created kms service
I20260620 18:37:41.868999 22676 mini_ranger_kms.cc:326] Created kudu user
I20260620 18:37:41.914484 22676 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 18:37:41 dist-test-slave-5lgp krb5kdc[1036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980659, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.22.37.20@KRBTEST.COM
I20260620 18:37:42.311874 22676 mini_ranger_kms.cc:384] Added ranger policy
I20260620 18:37:42.312199 22676 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/minicluster-data/ranger-kms:/tmp/dist-test-task1UXpXs/build/asan/bin/postgresql.jar:/tmp/dist-test-task1UXpXs/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task1UXpXs/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 18:37:42.312258 22676 mini_ranger_kms.cc:240] Using host: 127.22.37.20
I20260620 18:37:42.336727 22676 mini_ranger_kms.cc:276] Ranger KMS PID: 1429
I20260620 18:37:42.336934 22676 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.22.37.20:59949
Jun 20, 2026 6:37:43 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 6:37:43 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 6:37:43 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 6:37:43 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 6:37:44 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 6:37:44 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 18:37:44.666218 22676 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.416s user 0.092s sys 0.344s
I20260620 18:37:44.666460 22676 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 18:37:44.666677 22676 mini_ranger_kms.cc:401] 127.22.37.20:59949/kms/v1/keys
Jun 20 18:37:44 dist-test-slave-5lgp krb5kdc[1036](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781980664, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.22.37.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 18:37:45 dist-test-slave-5lgp krb5kdc[1036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980664, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.22.37.20@KRBTEST.COM for HTTP/127.22.37.20@KRBTEST.COM
Jun 20 18:37:45 dist-test-slave-5lgp krb5kdc[1036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980664, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.22.37.20@KRBTEST.COM for HTTP/127.22.37.20@KRBTEST.COM
Jun 20 18:37:47 dist-test-slave-5lgp krb5kdc[1036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980667, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.22.37.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.37.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.37.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab.
Entry for principal kudu/127.22.37.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab.
I20260620 18:37:47.828893 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.62:42795
--webserver_interface=127.22.37.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab
--principal=kudu/127.22.37.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:34125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.37.62:42795
--ranger_config_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.37.20:59949/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:37:48.169305 1480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:37:48.169744 1480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:37:48.169900 1480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:37:48.177141 1480 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:37:48.177289 1480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:37:48.177409 1480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:37:48.177524 1480 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:37:48.177608 1480 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:37:48.188336 1480 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:34125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/minicluster-data/master-0/wal
--ranger_kms_url=127.22.37.20:59949/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.22.37.62:42795
--ranger_config_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab
--principal=kudu/127.22.37.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.62:42795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.37.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:37:48.189777 1480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:37:48.191097 1480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:37:48.203493 1486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:48.204952 1488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:48.205111 1485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:37:48.205762 1480 server_base.cc:1034] running on GCE node
I20260620 18:37:48.206672 1480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:37:48.208437 1480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:37:48.209681 1480 hybrid_clock.cc:648] HybridClock initialized: now 1781980668209625 us; error 46 us; skew 500 ppm
I20260620 18:37:48.210075 1480 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:37:48 dist-test-slave-5lgp krb5kdc[1036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980668, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:37:48.217540 1480 init.cc:377] Logged in from keytab as kudu/127.22.37.62@KRBTEST.COM (short username kudu)
I20260620 18:37:48.219561 1480 webserver.cc:460] Webserver started at http://127.22.37.62:38759/ using document root <none> and password file <none>
I20260620 18:37:48.220250 1480 fs_manager.cc:359] Metadata directory not provided
I20260620 18:37:48.220355 1480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:37:48.220693 1480 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 18:37:48 dist-test-slave-5lgp krb5kdc[1036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980668, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.62@KRBTEST.COM for HTTP/127.22.37.20@KRBTEST.COM
I20260620 18:37:48.282572 1480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/minicluster-data/master-0/data/instance:
uuid: "0d37b1d9f74d41029b7ef81e5012d67f"
format_stamp: "Formatted at 2026-06-20 18:37:48 on dist-test-slave-5lgp"
server_key: "b95c6854df1e04fba38eee1cd527c2a6"
server_key_iv: "172d63bdb396fa5785adedac8bea1f89"
server_key_version: "kuduclusterkey@0"
I20260620 18:37:48.283413 1480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/minicluster-data/master-0/wal/instance:
uuid: "0d37b1d9f74d41029b7ef81e5012d67f"
format_stamp: "Formatted at 2026-06-20 18:37:48 on dist-test-slave-5lgp"
server_key: "b95c6854df1e04fba38eee1cd527c2a6"
server_key_iv: "172d63bdb396fa5785adedac8bea1f89"
server_key_version: "kuduclusterkey@0"
I20260620 18:37:48.288796 1480 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 18:37:48.305248 1495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:37:48.306838 1480 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 18:37:48.307018 1480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/minicluster-data/master-0/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/minicluster-data/master-0/wal
uuid: "0d37b1d9f74d41029b7ef81e5012d67f"
format_stamp: "Formatted at 2026-06-20 18:37:48 on dist-test-slave-5lgp"
server_key: "b95c6854df1e04fba38eee1cd527c2a6"
server_key_iv: "172d63bdb396fa5785adedac8bea1f89"
server_key_version: "kuduclusterkey@0"
I20260620 18:37:48.307322 1480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:37:48.339697 1480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:37:48.343637 1480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:37:48.343946 1480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:37:48.364578 1480 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.62:42795
I20260620 18:37:48.364598 1546 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.62:42795 every 8 connection(s)
I20260620 18:37:48.365991 1480 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
I20260620 18:37:48.368680 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 1480
I20260620 18:37:48.368893 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/minicluster-data/master-0/wal/instance
I20260620 18:37:48.369728 22676 external_mini_cluster.cc:1428] Setting key 9376427ef5342ed189a4c436ff0de88c
I20260620 18:37:48.370644 1547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:37:48.388367 1547 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d37b1d9f74d41029b7ef81e5012d67f: Bootstrap starting.
I20260620 18:37:48.392103 1547 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d37b1d9f74d41029b7ef81e5012d67f: Neither blocks nor log segments found. Creating new log.
I20260620 18:37:48.393144 1547 log.cc:824] T 00000000000000000000000000000000 P 0d37b1d9f74d41029b7ef81e5012d67f: Log is configured to *not* fsync() on all Append() calls
I20260620 18:37:48.396569 1547 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d37b1d9f74d41029b7ef81e5012d67f: No bootstrap required, opened a new log
Jun 20 18:37:48 dist-test-slave-5lgp krb5kdc[1036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980667, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM
I20260620 18:37:48.407235 1547 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d37b1d9f74d41029b7ef81e5012d67f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d37b1d9f74d41029b7ef81e5012d67f" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 42795 } }
I20260620 18:37:48.407703 1550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:48.371716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38046 (local address 127.22.37.62:42795)
0620 18:37:48.386731 (+ 15015us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:48.386747 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:48.386798 (+ 51us) server_negotiation.cc:407] Connection header received
0620 18:37:48.389173 (+ 2375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:48.389203 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:48.389742 (+ 539us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:48.390175 (+ 433us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:48.391740 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:48.393770 (+ 2030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:48.395321 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:48.396394 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:48.397473 (+ 1079us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:48.400452 (+ 2979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:48.400461 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:48.400475 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:37:48.400519 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:48.403577 (+ 3058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:48.404272 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:48.404279 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:48.404285 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:48.404426 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:48.404815 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:48.404820 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:48.404825 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:48.405415 (+ 590us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:48.405731 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:48.406185 (+ 454us) server_negotiation.cc:299] Negotiation successful
0620 18:37:48.406611 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14767,"thread_start_us":14532,"threads_started":1}
I20260620 18:37:48.407775 1547 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d37b1d9f74d41029b7ef81e5012d67f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:37:48.407886 1547 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d37b1d9f74d41029b7ef81e5012d67f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d37b1d9f74d41029b7ef81e5012d67f, State: Initialized, Role: FOLLOWER
I20260620 18:37:48.408582 1547 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d37b1d9f74d41029b7ef81e5012d67f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d37b1d9f74d41029b7ef81e5012d67f" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 42795 } }
I20260620 18:37:48.409204 1547 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d37b1d9f74d41029b7ef81e5012d67f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:37:48.409411 1547 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d37b1d9f74d41029b7ef81e5012d67f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:37:48.409674 1547 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d37b1d9f74d41029b7ef81e5012d67f [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:37:48.412982 1547 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d37b1d9f74d41029b7ef81e5012d67f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d37b1d9f74d41029b7ef81e5012d67f" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 42795 } }
I20260620 18:37:48.413549 1547 leader_election.cc:302] T 00000000000000000000000000000000 P 0d37b1d9f74d41029b7ef81e5012d67f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d37b1d9f74d41029b7ef81e5012d67f; no voters:
I20260620 18:37:48.414429 1547 leader_election.cc:288] T 00000000000000000000000000000000 P 0d37b1d9f74d41029b7ef81e5012d67f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:37:48.414538 1552 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d37b1d9f74d41029b7ef81e5012d67f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:37:48.415901 1552 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d37b1d9f74d41029b7ef81e5012d67f [term 1 LEADER]: Becoming Leader. State: Replica: 0d37b1d9f74d41029b7ef81e5012d67f, State: Running, Role: LEADER
I20260620 18:37:48.416510 1547 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d37b1d9f74d41029b7ef81e5012d67f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:37:48.416422 1552 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d37b1d9f74d41029b7ef81e5012d67f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d37b1d9f74d41029b7ef81e5012d67f" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 42795 } }
I20260620 18:37:48.426863 1553 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d37b1d9f74d41029b7ef81e5012d67f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d37b1d9f74d41029b7ef81e5012d67f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d37b1d9f74d41029b7ef81e5012d67f" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 42795 } } }
I20260620 18:37:48.427256 1553 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d37b1d9f74d41029b7ef81e5012d67f [sys.catalog]: This master's current role is: LEADER
I20260620 18:37:48.427698 1554 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d37b1d9f74d41029b7ef81e5012d67f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d37b1d9f74d41029b7ef81e5012d67f. Latest consensus state: current_term: 1 leader_uuid: "0d37b1d9f74d41029b7ef81e5012d67f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d37b1d9f74d41029b7ef81e5012d67f" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 42795 } } }
I20260620 18:37:48.427984 1554 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d37b1d9f74d41029b7ef81e5012d67f [sys.catalog]: This master's current role is: LEADER
I20260620 18:37:48.431802 1559 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:37:48.437772 1559 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:37:48.438519 1547 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 18:37:48.470424 1559 catalog_manager.cc:1347] Generated new cluster ID: 3dfb5f23dae24459a69e743153bfd842
I20260620 18:37:48.470582 1559 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:37:48.533929 1559 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:37:48.536433 1559 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:37:48.553390 1559 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d37b1d9f74d41029b7ef81e5012d67f: Generated new TSK 0
I20260620 18:37:48.554760 1559 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.37.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.37.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.37.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.1.keytab.
Entry for principal kudu/127.22.37.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.1.keytab.
I20260620 18:37:49.274247 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.37.20:59949/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.37.1:0
--local_ip_for_outbound_sockets=127.22.37.1
--webserver_interface=127.22.37.1
--webserver_port=0
--tserver_master_addrs=127.22.37.62:42795
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.1.keytab
--principal=kudu/127.22.37.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:34125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 18:37:49 dist-test-slave-5lgp krb5kdc[1036](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781980669, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 18:37:49.734529 1592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:37:49.735107 1592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:37:49.735313 1592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:37:49.747392 1592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:37:49.747649 1592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:37:49.747885 1592 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.37.1
I20260620 18:37:49.764456 1592 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:34125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.22.37.20:59949/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.1.keytab
--principal=kudu/127.22.37.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.37.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.37.1
--webserver_port=0
--tserver_master_addrs=127.22.37.62:42795
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.37.1
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:37:49.766368 1592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:37:49.767742 1592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:37:49.781948 1603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:49.782509 1602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:49.782686 1605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:37:49.782927 1592 server_base.cc:1034] running on GCE node
I20260620 18:37:49.783630 1592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:37:49.784688 1592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:37:49.785905 1592 hybrid_clock.cc:648] HybridClock initialized: now 1781980669785834 us; error 63 us; skew 500 ppm
I20260620 18:37:49.786254 1592 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:37:49 dist-test-slave-5lgp krb5kdc[1036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980669, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:37:49.790342 1592 init.cc:377] Logged in from keytab as kudu/127.22.37.1@KRBTEST.COM (short username kudu)
I20260620 18:37:49.792714 1592 webserver.cc:460] Webserver started at http://127.22.37.1:43873/ using document root <none> and password file <none>
I20260620 18:37:49.793848 1592 fs_manager.cc:359] Metadata directory not provided
I20260620 18:37:49.793994 1592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:37:49.794346 1592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 18:37:49 dist-test-slave-5lgp krb5kdc[1036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980669, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.1@KRBTEST.COM for HTTP/127.22.37.20@KRBTEST.COM
I20260620 18:37:49.814237 1592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/minicluster-data/ts-0/data/instance:
uuid: "1da850e274ca47e78d4760c19a8b1998"
format_stamp: "Formatted at 2026-06-20 18:37:49 on dist-test-slave-5lgp"
server_key: "ec8ec8bbed213d614a7c5048c477e1fc"
server_key_iv: "3df58036d4ea3b0081b7a114e030c6ea"
server_key_version: "kuduclusterkey@0"
I20260620 18:37:49.816356 1592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/minicluster-data/ts-0/wal/instance:
uuid: "1da850e274ca47e78d4760c19a8b1998"
format_stamp: "Formatted at 2026-06-20 18:37:49 on dist-test-slave-5lgp"
server_key: "ec8ec8bbed213d614a7c5048c477e1fc"
server_key_iv: "3df58036d4ea3b0081b7a114e030c6ea"
server_key_version: "kuduclusterkey@0"
I20260620 18:37:49.846390 1592 fs_manager.cc:691] Time spent creating directory manager: real 0.029s user 0.019s sys 0.008s
I20260620 18:37:49.881117 1612 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:37:49.883445 1592 fs_manager.cc:725] Time spent opening block manager: real 0.016s user 0.009s sys 0.006s
I20260620 18:37:49.883656 1592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/minicluster-data/ts-0/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/minicluster-data/ts-0/wal
uuid: "1da850e274ca47e78d4760c19a8b1998"
format_stamp: "Formatted at 2026-06-20 18:37:49 on dist-test-slave-5lgp"
server_key: "ec8ec8bbed213d614a7c5048c477e1fc"
server_key_iv: "3df58036d4ea3b0081b7a114e030c6ea"
server_key_version: "kuduclusterkey@0"
I20260620 18:37:49.883896 1592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:37:49.908008 1592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:37:49.912298 1592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:37:49.912587 1592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:37:49.913599 1592 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:37:49.915045 1592 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:37:49.915143 1592 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:37:49.915290 1592 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:37:49.915373 1592 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 18:37:49 dist-test-slave-5lgp krb5kdc[1036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980669, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.62@KRBTEST.COM for HTTP/127.22.37.20@KRBTEST.COM
I20260620 18:37:50.130359 1592 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.1:45681
I20260620 18:37:50.130419 1724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.1:45681 every 8 connection(s)
I20260620 18:37:50.132020 1592 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/minicluster-data/ts-0/data/info.pb
I20260620 18:37:50.136915 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 1592
I20260620 18:37:50.137153 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/minicluster-data/ts-0/wal/instance
I20260620 18:37:50.138059 22676 external_mini_cluster.cc:1428] Setting key c6a4e291c70b174b60567a62ee5dcbd6
Jun 20 18:37:50 dist-test-slave-5lgp krb5kdc[1036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980669, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.1@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM
I20260620 18:37:50.173911 1728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:50.136237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.1:43611 (local address 127.22.37.62:42795)
0620 18:37:50.149797 (+ 13560us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:50.149802 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:50.149820 (+ 18us) server_negotiation.cc:407] Connection header received
0620 18:37:50.150688 (+ 868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:50.150694 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:50.150776 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:50.150914 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:50.152978 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:50.154145 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:50.156568 (+ 2423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:50.157227 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:50.157322 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:50.166487 (+ 9165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:50.166494 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:50.166499 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:37:50.166543 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:50.170088 (+ 3545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:50.171102 (+ 1014us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:50.171107 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:50.171111 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:50.171271 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:50.171795 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:50.171800 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:50.171804 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:50.172141 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:50.172261 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:50.173263 (+ 1002us) server_negotiation.cc:299] Negotiation successful
0620 18:37:50.173519 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13431,"thread_start_us":13313,"threads_started":1}
I20260620 18:37:50.176487 1727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:50.135629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:42795 (local address 127.22.37.1:43611)
0620 18:37:50.145996 (+ 10367us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:50.146046 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 18:37:50.149320 (+ 3274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:37:50.151334 (+ 2014us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:37:50.151347 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:37:50.151984 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:37:50.152762 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:50.152786 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:50.154362 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:50.154367 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:50.156397 (+ 2030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:50.156413 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:50.157680 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:50.157686 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:50.157985 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:50.158805 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:37:50.158840 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:37:50.166166 (+ 7326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:37:50.170280 (+ 4114us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:37:50.170291 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:37:50.170313 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:37:50.170956 (+ 643us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:37:50.171425 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:37:50.171430 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:37:50.171433 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:37:50.171671 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:37:50.172364 (+ 693us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:37:50.172371 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:37:50.172819 (+ 448us) client_negotiation.cc:769] Sending connection context
0620 18:37:50.174480 (+ 1661us) client_negotiation.cc:240] Negotiation successful
0620 18:37:50.175050 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10053,"thread_start_us":9822,"threads_started":1}
WARNING: no policy specified for kudu/127.22.37.2@KRBTEST.COM; defaulting to no policy
I20260620 18:37:50.178661 1725 heartbeater.cc:347] Connected to a master server at 127.22.37.62:42795
I20260620 18:37:50.179162 1725 heartbeater.cc:464] Registering TS with master...
I20260620 18:37:50.180048 1725 heartbeater.cc:511] Master 127.22.37.62:42795 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.37.2@KRBTEST.COM" created.
I20260620 18:37:50.182664 1512 ts_manager.cc:194] Registered new tserver with Master: 1da850e274ca47e78d4760c19a8b1998 (127.22.37.1:45681)
I20260620 18:37:50.184540 1512 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.37.1@KRBTEST.COM'} at 127.22.37.1:43611
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.37.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.2.keytab.
Entry for principal kudu/127.22.37.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.2.keytab.
I20260620 18:37:50.219655 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.37.20:59949/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.37.2:0
--local_ip_for_outbound_sockets=127.22.37.2
--webserver_interface=127.22.37.2
--webserver_port=0
--tserver_master_addrs=127.22.37.62:42795
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.2.keytab
--principal=kudu/127.22.37.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:34125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 18:37:50 dist-test-slave-5lgp krb5kdc[1036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980669, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.62@KRBTEST.COM for HTTP/127.22.37.20@KRBTEST.COM
W20260620 18:37:50.694574 1731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:37:50.694844 1731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:37:50.694936 1731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:37:50.702649 1731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:37:50.702756 1731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:37:50.702868 1731 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.37.2
I20260620 18:37:50.712433 1731 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:34125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.22.37.20:59949/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.2.keytab
--principal=kudu/127.22.37.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.37.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.37.2
--webserver_port=0
--tserver_master_addrs=127.22.37.62:42795
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.37.2
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:37:50.713740 1731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:37:50.714963 1731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:37:50.728034 1745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:50.728048 1748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:50.728212 1746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:37:50.729219 1731 server_base.cc:1034] running on GCE node
I20260620 18:37:50.729831 1731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:37:50.730671 1731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:37:50.731859 1731 hybrid_clock.cc:648] HybridClock initialized: now 1781980670731800 us; error 54 us; skew 500 ppm
I20260620 18:37:50.732125 1731 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:37:50 dist-test-slave-5lgp krb5kdc[1036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980670, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:37:50.735129 1731 init.cc:377] Logged in from keytab as kudu/127.22.37.2@KRBTEST.COM (short username kudu)
I20260620 18:37:50.737015 1731 webserver.cc:460] Webserver started at http://127.22.37.2:44027/ using document root <none> and password file <none>
I20260620 18:37:50.737748 1731 fs_manager.cc:359] Metadata directory not provided
I20260620 18:37:50.737895 1731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:37:50.738198 1731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 18:37:50 dist-test-slave-5lgp krb5kdc[1036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980670, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.2@KRBTEST.COM for HTTP/127.22.37.20@KRBTEST.COM
I20260620 18:37:50.754061 1731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/minicluster-data/ts-1/data/instance:
uuid: "6dabe2324f6145ee95783fa302c6ba71"
format_stamp: "Formatted at 2026-06-20 18:37:50 on dist-test-slave-5lgp"
server_key: "fa33073ff61d9007b3461c66ceb1ab0b"
server_key_iv: "12dd1f28c536e376e2cee2b0458d3b3f"
server_key_version: "kuduclusterkey@0"
I20260620 18:37:50.754944 1731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/minicluster-data/ts-1/wal/instance:
uuid: "6dabe2324f6145ee95783fa302c6ba71"
format_stamp: "Formatted at 2026-06-20 18:37:50 on dist-test-slave-5lgp"
server_key: "fa33073ff61d9007b3461c66ceb1ab0b"
server_key_iv: "12dd1f28c536e376e2cee2b0458d3b3f"
server_key_version: "kuduclusterkey@0"
I20260620 18:37:50.760608 1731 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260620 18:37:50.781018 1755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:37:50.783043 1731 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.003s sys 0.004s
I20260620 18:37:50.783227 1731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/minicluster-data/ts-1/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/minicluster-data/ts-1/wal
uuid: "6dabe2324f6145ee95783fa302c6ba71"
format_stamp: "Formatted at 2026-06-20 18:37:50 on dist-test-slave-5lgp"
server_key: "fa33073ff61d9007b3461c66ceb1ab0b"
server_key_iv: "12dd1f28c536e376e2cee2b0458d3b3f"
server_key_version: "kuduclusterkey@0"
I20260620 18:37:50.783486 1731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:37:50.798352 1731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:37:50.802297 1731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:37:50.802647 1731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:37:50.803678 1731 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:37:50.805047 1731 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:37:50.805119 1731 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:37:50.805284 1731 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:37:50.805336 1731 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:37:50.849171 1731 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.2:37791
I20260620 18:37:50.849188 1867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.2:37791 every 8 connection(s)
I20260620 18:37:50.850544 1731 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/minicluster-data/ts-1/data/info.pb
I20260620 18:37:50.851943 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 1731
I20260620 18:37:50.852114 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/minicluster-data/ts-1/wal/instance
I20260620 18:37:50.852694 22676 external_mini_cluster.cc:1428] Setting key d0192d15dc37ba2d996c364ce49b8121
Jun 20 18:37:50 dist-test-slave-5lgp krb5kdc[1036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980670, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.2@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.37.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.37.3@KRBTEST.COM" created.
I20260620 18:37:50.881026 1870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:50.854293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.2:46749 (local address 127.22.37.62:42795)
0620 18:37:50.861819 (+ 7526us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:50.861828 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:50.861850 (+ 22us) server_negotiation.cc:407] Connection header received
0620 18:37:50.861913 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:50.861921 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:50.862042 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:50.862158 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:50.864231 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:50.865443 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:50.867831 (+ 2388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:50.868592 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:50.868733 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:50.873303 (+ 4570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:50.873310 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:50.873315 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:37:50.873357 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:50.877504 (+ 4147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:50.878477 (+ 973us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:50.878484 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:50.878488 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:50.878638 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:50.879114 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:50.879119 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:50.879123 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:50.879487 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:50.879619 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:50.880313 (+ 694us) server_negotiation.cc:299] Negotiation successful
0620 18:37:50.880595 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7371,"thread_start_us":7236,"threads_started":1}
I20260620 18:37:50.886621 1871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:50.854775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:42795 (local address 127.22.37.2:46749)
0620 18:37:50.856282 (+ 1507us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:50.856334 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 18:37:50.859033 (+ 2699us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:37:50.862509 (+ 3476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:37:50.862522 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:37:50.863167 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:37:50.864013 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:50.864040 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:50.865621 (+ 1581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:50.865627 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:50.867656 (+ 2029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:50.867671 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:50.868854 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:50.868864 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:50.869188 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:50.870055 (+ 867us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:37:50.870085 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:37:50.873035 (+ 2950us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:37:50.877682 (+ 4647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:37:50.877693 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:37:50.877713 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:37:50.878352 (+ 639us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:37:50.878750 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:37:50.878754 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:37:50.878756 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:37:50.879022 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:37:50.879609 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:37:50.879615 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:37:50.879944 (+ 329us) client_negotiation.cc:769] Sending connection context
0620 18:37:50.880193 (+ 249us) client_negotiation.cc:240] Negotiation successful
0620 18:37:50.880584 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1175,"thread_start_us":1064,"threads_started":1}
I20260620 18:37:50.888448 1868 heartbeater.cc:347] Connected to a master server at 127.22.37.62:42795
I20260620 18:37:50.888793 1868 heartbeater.cc:464] Registering TS with master...
I20260620 18:37:50.889704 1868 heartbeater.cc:511] Master 127.22.37.62:42795 requested a full tablet report, sending...
I20260620 18:37:50.891351 1512 ts_manager.cc:194] Registered new tserver with Master: 6dabe2324f6145ee95783fa302c6ba71 (127.22.37.2:37791)
I20260620 18:37:50.892674 1512 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.37.2@KRBTEST.COM'} at 127.22.37.2:46749
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.37.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.3.keytab.
Entry for principal kudu/127.22.37.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.3.keytab.
I20260620 18:37:50.909667 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.37.20:59949/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.37.3:0
--local_ip_for_outbound_sockets=127.22.37.3
--webserver_interface=127.22.37.3
--webserver_port=0
--tserver_master_addrs=127.22.37.62:42795
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.3.keytab
--principal=kudu/127.22.37.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:34125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:37:51.190305 1725 heartbeater.cc:503] Master 127.22.37.62:42795 was elected leader, sending a full tablet report...
W20260620 18:37:51.254377 1874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:37:51.254704 1874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:37:51.254882 1874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:37:51.261991 1874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:37:51.262137 1874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:37:51.262322 1874 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.37.3
I20260620 18:37:51.272277 1874 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:34125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.22.37.20:59949/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.3.keytab
--principal=kudu/127.22.37.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.37.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.37.3
--webserver_port=0
--tserver_master_addrs=127.22.37.62:42795
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.37.3
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:37:51.273599 1874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:37:51.274905 1874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:37:51.288702 1882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:51.288741 1880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:51.288894 1879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:37:51.289777 1874 server_base.cc:1034] running on GCE node
I20260620 18:37:51.290364 1874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:37:51.291282 1874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:37:51.292531 1874 hybrid_clock.cc:648] HybridClock initialized: now 1781980671292476 us; error 62 us; skew 500 ppm
I20260620 18:37:51.292886 1874 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:37:51 dist-test-slave-5lgp krb5kdc[1036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980671, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:37:51.295976 1874 init.cc:377] Logged in from keytab as kudu/127.22.37.3@KRBTEST.COM (short username kudu)
I20260620 18:37:51.297338 1874 webserver.cc:460] Webserver started at http://127.22.37.3:44739/ using document root <none> and password file <none>
I20260620 18:37:51.298123 1874 fs_manager.cc:359] Metadata directory not provided
I20260620 18:37:51.298269 1874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:37:51.298568 1874 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 18:37:51 dist-test-slave-5lgp krb5kdc[1036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980671, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.3@KRBTEST.COM for HTTP/127.22.37.20@KRBTEST.COM
I20260620 18:37:51.313735 1874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/minicluster-data/ts-2/data/instance:
uuid: "c03d239d0b234c5a9c19eaf8551526c5"
format_stamp: "Formatted at 2026-06-20 18:37:51 on dist-test-slave-5lgp"
server_key: "af57e1e12205c142557095801b5b5697"
server_key_iv: "66250901a88d9fa554a16e3845b3f20b"
server_key_version: "kuduclusterkey@0"
I20260620 18:37:51.314785 1874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/minicluster-data/ts-2/wal/instance:
uuid: "c03d239d0b234c5a9c19eaf8551526c5"
format_stamp: "Formatted at 2026-06-20 18:37:51 on dist-test-slave-5lgp"
server_key: "af57e1e12205c142557095801b5b5697"
server_key_iv: "66250901a88d9fa554a16e3845b3f20b"
server_key_version: "kuduclusterkey@0"
I20260620 18:37:51.320470 1874 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:37:51.335146 1889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:37:51.336721 1874 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:37:51.336915 1874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/minicluster-data/ts-2/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/minicluster-data/ts-2/wal
uuid: "c03d239d0b234c5a9c19eaf8551526c5"
format_stamp: "Formatted at 2026-06-20 18:37:51 on dist-test-slave-5lgp"
server_key: "af57e1e12205c142557095801b5b5697"
server_key_iv: "66250901a88d9fa554a16e3845b3f20b"
server_key_version: "kuduclusterkey@0"
I20260620 18:37:51.337147 1874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:37:51.363843 1874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:37:51.367734 1874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:37:51.368299 1874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:37:51.369287 1874 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:37:51.370680 1874 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:37:51.370793 1874 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:37:51.370946 1874 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:37:51.371018 1874 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:37:51.406309 1874 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.3:44953
I20260620 18:37:51.406335 2001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.3:44953 every 8 connection(s)
I20260620 18:37:51.407639 1874 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/minicluster-data/ts-2/data/info.pb
I20260620 18:37:51.415426 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 1874
I20260620 18:37:51.415637 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/minicluster-data/ts-2/wal/instance
I20260620 18:37:51.416199 22676 external_mini_cluster.cc:1428] Setting key 857dcbcb082feb687f5abfaa31717cbd
Jun 20 18:37:51 dist-test-slave-5lgp krb5kdc[1036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980671, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.3@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM
I20260620 18:37:51.428413 2004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:51.410247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.3:50347 (local address 127.22.37.62:42795)
0620 18:37:51.411752 (+ 1505us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:51.411761 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:51.412230 (+ 469us) server_negotiation.cc:407] Connection header received
0620 18:37:51.414101 (+ 1871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:51.414105 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:51.414182 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:51.414282 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:51.416034 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:51.416958 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:51.418843 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:51.419295 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:51.419429 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:51.422613 (+ 3184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:51.422618 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:51.422622 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:37:51.422651 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:51.425286 (+ 2635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:51.425994 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:51.425998 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:51.426000 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:51.426114 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:51.426583 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:51.426586 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:51.426588 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:51.427220 (+ 632us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:51.427325 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:51.427993 (+ 668us) server_negotiation.cc:299] Negotiation successful
0620 18:37:51.428148 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1378,"thread_start_us":1235,"threads_started":1}
I20260620 18:37:51.429425 2005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:51.411089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:42795 (local address 127.22.37.3:50347)
0620 18:37:51.412075 (+ 986us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:51.412121 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 18:37:51.413854 (+ 1733us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:37:51.414450 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:37:51.414458 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:37:51.414950 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:37:51.415840 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:51.415880 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:51.417118 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:51.417122 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:51.418688 (+ 1566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:51.418707 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:51.419521 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:51.419527 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:51.419760 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:51.420384 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:37:51.420416 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:37:51.422435 (+ 2019us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:37:51.425454 (+ 3019us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:37:51.425461 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:37:51.425474 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:37:51.425892 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:37:51.426238 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:37:51.426243 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:37:51.426246 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:37:51.426474 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:37:51.427337 (+ 863us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:37:51.427343 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:37:51.427669 (+ 326us) client_negotiation.cc:769] Sending connection context
0620 18:37:51.427902 (+ 233us) client_negotiation.cc:240] Negotiation successful
0620 18:37:51.428283 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":689,"thread_start_us":407,"threads_started":1}
I20260620 18:37:51.430872 2002 heartbeater.cc:347] Connected to a master server at 127.22.37.62:42795
I20260620 18:37:51.431188 2002 heartbeater.cc:464] Registering TS with master...
I20260620 18:37:51.432052 2002 heartbeater.cc:511] Master 127.22.37.62:42795 requested a full tablet report, sending...
I20260620 18:37:51.433557 1511 ts_manager.cc:194] Registered new tserver with Master: c03d239d0b234c5a9c19eaf8551526c5 (127.22.37.3:44953)
I20260620 18:37:51.434463 1511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.37.3@KRBTEST.COM'} at 127.22.37.3:50347
I20260620 18:37:51.443909 22676 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:37:51.460281 2004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:51.448316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38060 (local address 127.22.37.62:42795)
0620 18:37:51.448498 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:51.448503 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:51.449017 (+ 514us) server_negotiation.cc:407] Connection header received
0620 18:37:51.449127 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:51.449131 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:51.449194 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:51.449276 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:51.450647 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:51.451377 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:51.452660 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:51.453064 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:51.453152 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:51.455534 (+ 2382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:51.455539 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:51.455542 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:37:51.455571 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:51.458331 (+ 2760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:51.458868 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:51.458872 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:51.458874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:51.458979 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:51.459271 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:51.459274 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:51.459276 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:51.459513 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:51.459683 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:51.459909 (+ 226us) server_negotiation.cc:299] Negotiation successful
0620 18:37:51.460061 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 18:37:51.464102 1511 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38060:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:37:51.470932 1511 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:37:51.497416 2017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:51.483880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.3:44953 (local address 127.0.0.1:48934)
0620 18:37:51.484733 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:51.484754 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 18:37:51.484873 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:37:51.486420 (+ 1547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:37:51.486432 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:37:51.486473 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:37:51.486994 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:51.487010 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:51.490912 (+ 3902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:51.490918 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:51.494318 (+ 3400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:51.494330 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:51.496213 (+ 1883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:51.496219 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:51.496410 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:51.496466 (+ 56us) client_negotiation.cc:769] Sending connection context
0620 18:37:51.496581 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 18:37:51.496756 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":730,"thread_start_us":1096,"threads_started":1}
I20260620 18:37:51.497813 2019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:51.484508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48934 (local address 127.22.37.3:44953)
0620 18:37:51.485594 (+ 1086us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:51.485646 (+ 52us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:51.485689 (+ 43us) server_negotiation.cc:407] Connection header received
0620 18:37:51.485831 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:51.485842 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:51.486142 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:51.487308 (+ 1166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:37:51.487659 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:51.489888 (+ 2229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:51.494548 (+ 4660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:51.495988 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:51.496944 (+ 956us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:51.497191 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:51.497257 (+ 66us) server_negotiation.cc:299] Negotiation successful
0620 18:37:51.497495 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":931,"thread_start_us":859,"threads_started":1}
I20260620 18:37:51.501416 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:51.483147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.2:37791 (local address 127.0.0.1:57184)
0620 18:37:51.483902 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:51.483946 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 18:37:51.484118 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:37:51.491975 (+ 7857us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:37:51.491980 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:37:51.492010 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:37:51.492543 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:51.492556 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:51.495552 (+ 2996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:51.495558 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:51.498870 (+ 3312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:51.498884 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:51.500621 (+ 1737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:51.500629 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:51.500808 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:51.500838 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 18:37:51.500927 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 18:37:51.501042 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"mutex_wait_us":586,"thread_start_us":319,"threads_started":1}
I20260620 18:37:51.501968 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:51.483291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57184 (local address 127.22.37.2:37791)
0620 18:37:51.491347 (+ 8056us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:51.491355 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:51.491380 (+ 25us) server_negotiation.cc:407] Connection header received
0620 18:37:51.491469 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:51.491479 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:51.491718 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:51.492640 (+ 922us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:37:51.493996 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:51.495448 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:51.499059 (+ 3611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:51.500403 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:51.500530 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:51.500721 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:51.501501 (+ 780us) server_negotiation.cc:299] Negotiation successful
0620 18:37:51.501665 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7902,"thread_start_us":9503,"threads_started":1}
I20260620 18:37:51.502518 1937 tablet_service.cc:1467] Processing CreateTablet for tablet a931a8da37a645439f59a12eec52c35c (DEFAULT_TABLE table=test-table [id=d2b755d7dadf4ad59ef704fd357b5967]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:37:51.503261 2018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:51.484467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.1:45681 (local address 127.0.0.1:50710)
0620 18:37:51.490715 (+ 6248us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:51.490731 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:37:51.490912 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:37:51.492206 (+ 1294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:37:51.492212 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:37:51.492244 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:37:51.493463 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:51.493480 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:51.498092 (+ 4612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:51.498099 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:51.500571 (+ 2472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:51.500584 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:51.502558 (+ 1974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:51.502564 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:51.502728 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:51.502759 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 18:37:51.502836 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 18:37:51.502959 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6018,"mutex_wait_us":6,"thread_start_us":5697,"threads_started":1}
I20260620 18:37:51.503865 2020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:51.485164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50710 (local address 127.22.37.1:45681)
0620 18:37:51.490919 (+ 5755us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:51.490928 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:51.490956 (+ 28us) server_negotiation.cc:407] Connection header received
0620 18:37:51.491176 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:51.491187 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:51.491540 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:51.491808 (+ 268us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:37:51.493659 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:51.495336 (+ 1677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:51.500735 (+ 5399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:51.502086 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:51.502208 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:51.502427 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:51.503502 (+ 1075us) server_negotiation.cc:299] Negotiation successful
0620 18:37:51.503630 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5591,"thread_start_us":5434,"threads_started":1}
I20260620 18:37:51.503994 1937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a931a8da37a645439f59a12eec52c35c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:37:51.505200 1803 tablet_service.cc:1467] Processing CreateTablet for tablet a931a8da37a645439f59a12eec52c35c (DEFAULT_TABLE table=test-table [id=d2b755d7dadf4ad59ef704fd357b5967]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:37:51.506510 1803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a931a8da37a645439f59a12eec52c35c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:37:51.507112 1660 tablet_service.cc:1467] Processing CreateTablet for tablet a931a8da37a645439f59a12eec52c35c (DEFAULT_TABLE table=test-table [id=d2b755d7dadf4ad59ef704fd357b5967]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:37:51.508109 1660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a931a8da37a645439f59a12eec52c35c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:37:51.518564 2021 tablet_bootstrap.cc:492] T a931a8da37a645439f59a12eec52c35c P c03d239d0b234c5a9c19eaf8551526c5: Bootstrap starting.
I20260620 18:37:51.520135 2022 tablet_bootstrap.cc:492] T a931a8da37a645439f59a12eec52c35c P 1da850e274ca47e78d4760c19a8b1998: Bootstrap starting.
I20260620 18:37:51.522277 2021 tablet_bootstrap.cc:654] T a931a8da37a645439f59a12eec52c35c P c03d239d0b234c5a9c19eaf8551526c5: Neither blocks nor log segments found. Creating new log.
I20260620 18:37:51.523521 2021 log.cc:824] T a931a8da37a645439f59a12eec52c35c P c03d239d0b234c5a9c19eaf8551526c5: Log is configured to *not* fsync() on all Append() calls
I20260620 18:37:51.525352 2022 tablet_bootstrap.cc:654] T a931a8da37a645439f59a12eec52c35c P 1da850e274ca47e78d4760c19a8b1998: Neither blocks nor log segments found. Creating new log.
I20260620 18:37:51.526053 2023 tablet_bootstrap.cc:492] T a931a8da37a645439f59a12eec52c35c P 6dabe2324f6145ee95783fa302c6ba71: Bootstrap starting.
I20260620 18:37:51.526490 2021 tablet_bootstrap.cc:492] T a931a8da37a645439f59a12eec52c35c P c03d239d0b234c5a9c19eaf8551526c5: No bootstrap required, opened a new log
I20260620 18:37:51.526775 2021 ts_tablet_manager.cc:1397] T a931a8da37a645439f59a12eec52c35c P c03d239d0b234c5a9c19eaf8551526c5: Time spent bootstrapping tablet: real 0.008s user 0.008s sys 0.000s
I20260620 18:37:51.526726 2022 log.cc:824] T a931a8da37a645439f59a12eec52c35c P 1da850e274ca47e78d4760c19a8b1998: Log is configured to *not* fsync() on all Append() calls
I20260620 18:37:51.529544 2022 tablet_bootstrap.cc:492] T a931a8da37a645439f59a12eec52c35c P 1da850e274ca47e78d4760c19a8b1998: No bootstrap required, opened a new log
I20260620 18:37:51.529798 2022 ts_tablet_manager.cc:1397] T a931a8da37a645439f59a12eec52c35c P 1da850e274ca47e78d4760c19a8b1998: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.003s
I20260620 18:37:51.529822 2023 tablet_bootstrap.cc:654] T a931a8da37a645439f59a12eec52c35c P 6dabe2324f6145ee95783fa302c6ba71: Neither blocks nor log segments found. Creating new log.
I20260620 18:37:51.530959 2023 log.cc:824] T a931a8da37a645439f59a12eec52c35c P 6dabe2324f6145ee95783fa302c6ba71: Log is configured to *not* fsync() on all Append() calls
I20260620 18:37:51.535260 2023 tablet_bootstrap.cc:492] T a931a8da37a645439f59a12eec52c35c P 6dabe2324f6145ee95783fa302c6ba71: No bootstrap required, opened a new log
I20260620 18:37:51.535514 2023 ts_tablet_manager.cc:1397] T a931a8da37a645439f59a12eec52c35c P 6dabe2324f6145ee95783fa302c6ba71: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260620 18:37:51.537714 2022 raft_consensus.cc:348] T a931a8da37a645439f59a12eec52c35c P 1da850e274ca47e78d4760c19a8b1998 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dabe2324f6145ee95783fa302c6ba71" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 37791 } } peers { permanent_uuid: "1da850e274ca47e78d4760c19a8b1998" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 45681 } } peers { permanent_uuid: "c03d239d0b234c5a9c19eaf8551526c5" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 44953 } }
I20260620 18:37:51.538089 2022 raft_consensus.cc:374] T a931a8da37a645439f59a12eec52c35c P 1da850e274ca47e78d4760c19a8b1998 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:37:51.538244 2022 raft_consensus.cc:729] T a931a8da37a645439f59a12eec52c35c P 1da850e274ca47e78d4760c19a8b1998 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1da850e274ca47e78d4760c19a8b1998, State: Initialized, Role: FOLLOWER
I20260620 18:37:51.538172 2021 raft_consensus.cc:348] T a931a8da37a645439f59a12eec52c35c P c03d239d0b234c5a9c19eaf8551526c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dabe2324f6145ee95783fa302c6ba71" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 37791 } } peers { permanent_uuid: "1da850e274ca47e78d4760c19a8b1998" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 45681 } } peers { permanent_uuid: "c03d239d0b234c5a9c19eaf8551526c5" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 44953 } }
I20260620 18:37:51.538671 2021 raft_consensus.cc:374] T a931a8da37a645439f59a12eec52c35c P c03d239d0b234c5a9c19eaf8551526c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:37:51.538770 2021 raft_consensus.cc:729] T a931a8da37a645439f59a12eec52c35c P c03d239d0b234c5a9c19eaf8551526c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c03d239d0b234c5a9c19eaf8551526c5, State: Initialized, Role: FOLLOWER
I20260620 18:37:51.538940 2022 consensus_queue.cc:260] T a931a8da37a645439f59a12eec52c35c P 1da850e274ca47e78d4760c19a8b1998 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dabe2324f6145ee95783fa302c6ba71" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 37791 } } peers { permanent_uuid: "1da850e274ca47e78d4760c19a8b1998" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 45681 } } peers { permanent_uuid: "c03d239d0b234c5a9c19eaf8551526c5" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 44953 } }
I20260620 18:37:51.539319 2021 consensus_queue.cc:260] T a931a8da37a645439f59a12eec52c35c P c03d239d0b234c5a9c19eaf8551526c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dabe2324f6145ee95783fa302c6ba71" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 37791 } } peers { permanent_uuid: "1da850e274ca47e78d4760c19a8b1998" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 45681 } } peers { permanent_uuid: "c03d239d0b234c5a9c19eaf8551526c5" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 44953 } }
I20260620 18:37:51.540725 2022 ts_tablet_manager.cc:1428] T a931a8da37a645439f59a12eec52c35c P 1da850e274ca47e78d4760c19a8b1998: Time spent starting tablet: real 0.011s user 0.009s sys 0.001s
I20260620 18:37:51.545739 2002 heartbeater.cc:503] Master 127.22.37.62:42795 was elected leader, sending a full tablet report...
I20260620 18:37:51.545984 2023 raft_consensus.cc:348] T a931a8da37a645439f59a12eec52c35c P 6dabe2324f6145ee95783fa302c6ba71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dabe2324f6145ee95783fa302c6ba71" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 37791 } } peers { permanent_uuid: "1da850e274ca47e78d4760c19a8b1998" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 45681 } } peers { permanent_uuid: "c03d239d0b234c5a9c19eaf8551526c5" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 44953 } }
I20260620 18:37:51.546350 2023 raft_consensus.cc:374] T a931a8da37a645439f59a12eec52c35c P 6dabe2324f6145ee95783fa302c6ba71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:37:51.546432 2023 raft_consensus.cc:729] T a931a8da37a645439f59a12eec52c35c P 6dabe2324f6145ee95783fa302c6ba71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dabe2324f6145ee95783fa302c6ba71, State: Initialized, Role: FOLLOWER
I20260620 18:37:51.546854 2021 ts_tablet_manager.cc:1428] T a931a8da37a645439f59a12eec52c35c P c03d239d0b234c5a9c19eaf8551526c5: Time spent starting tablet: real 0.020s user 0.018s sys 0.001s
I20260620 18:37:51.546973 2023 consensus_queue.cc:260] T a931a8da37a645439f59a12eec52c35c P 6dabe2324f6145ee95783fa302c6ba71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dabe2324f6145ee95783fa302c6ba71" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 37791 } } peers { permanent_uuid: "1da850e274ca47e78d4760c19a8b1998" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 45681 } } peers { permanent_uuid: "c03d239d0b234c5a9c19eaf8551526c5" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 44953 } }
I20260620 18:37:51.548378 1868 heartbeater.cc:503] Master 127.22.37.62:42795 was elected leader, sending a full tablet report...
I20260620 18:37:51.548959 2023 ts_tablet_manager.cc:1428] T a931a8da37a645439f59a12eec52c35c P 6dabe2324f6145ee95783fa302c6ba71: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
W20260620 18:37:51.607955 1869 tablet.cc:2369] T a931a8da37a645439f59a12eec52c35c P 6dabe2324f6145ee95783fa302c6ba71: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:37:51.651654 1726 tablet.cc:2369] T a931a8da37a645439f59a12eec52c35c P 1da850e274ca47e78d4760c19a8b1998: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:37:51.659250 2003 tablet.cc:2369] T a931a8da37a645439f59a12eec52c35c P c03d239d0b234c5a9c19eaf8551526c5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:37:51.804603 2029 raft_consensus.cc:482] T a931a8da37a645439f59a12eec52c35c P 6dabe2324f6145ee95783fa302c6ba71 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:37:51.804962 2029 raft_consensus.cc:504] T a931a8da37a645439f59a12eec52c35c P 6dabe2324f6145ee95783fa302c6ba71 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dabe2324f6145ee95783fa302c6ba71" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 37791 } } peers { permanent_uuid: "1da850e274ca47e78d4760c19a8b1998" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 45681 } } peers { permanent_uuid: "c03d239d0b234c5a9c19eaf8551526c5" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 44953 } }
I20260620 18:37:51.806663 2029 leader_election.cc:288] T a931a8da37a645439f59a12eec52c35c P 6dabe2324f6145ee95783fa302c6ba71 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1da850e274ca47e78d4760c19a8b1998 (127.22.37.1:45681), c03d239d0b234c5a9c19eaf8551526c5 (127.22.37.3:44953)
I20260620 18:37:51.813715 2020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:51.807081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.2:53553 (local address 127.22.37.1:45681)
0620 18:37:51.807489 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:51.807497 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:51.807763 (+ 266us) server_negotiation.cc:407] Connection header received
0620 18:37:51.808005 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:51.808009 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:51.808065 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:51.808157 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:37:51.808707 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:51.809489 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:51.811863 (+ 2374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:51.812688 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:51.812792 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:51.812866 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:51.813302 (+ 436us) server_negotiation.cc:299] Negotiation successful
0620 18:37:51.813452 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254}
I20260620 18:37:51.813771 2019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:51.807262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.2:37923 (local address 127.22.37.3:44953)
0620 18:37:51.807473 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:51.807480 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:51.808052 (+ 572us) server_negotiation.cc:407] Connection header received
0620 18:37:51.808234 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:51.808239 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:51.808301 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:51.808403 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:37:51.808915 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:51.809784 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:51.811994 (+ 2210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:51.812877 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:51.812997 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:51.813097 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:51.813395 (+ 298us) server_negotiation.cc:299] Negotiation successful
0620 18:37:51.813546 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 18:37:51.813838 2030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:51.806898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.1:45681 (local address 127.22.37.2:53553)
0620 18:37:51.807652 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:51.807668 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:37:51.807854 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:37:51.808182 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:37:51.808187 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:37:51.808211 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:37:51.808563 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:51.808571 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:51.809628 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:51.809633 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:51.811719 (+ 2086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:51.811730 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:51.812900 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:51.812908 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:51.813103 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:51.813151 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 18:37:51.813220 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 18:37:51.813326 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"mutex_wait_us":4,"thread_start_us":459,"threads_started":1}
I20260620 18:37:51.814338 2031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:51.807084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.3:44953 (local address 127.22.37.2:37923)
0620 18:37:51.807944 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:51.807957 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 18:37:51.808096 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:37:51.808409 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:37:51.808413 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:37:51.808438 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:37:51.808759 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:51.808767 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:51.809922 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:51.809926 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:51.811814 (+ 1888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:51.811830 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:51.813062 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:51.813066 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:51.813189 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:51.813242 (+ 53us) client_negotiation.cc:769] Sending connection context
0620 18:37:51.813334 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 18:37:51.813668 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":758,"mutex_wait_us":52,"thread_start_us":565,"threads_started":1}
I20260620 18:37:51.814859 1680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a931a8da37a645439f59a12eec52c35c" candidate_uuid: "6dabe2324f6145ee95783fa302c6ba71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1da850e274ca47e78d4760c19a8b1998" is_pre_election: true
I20260620 18:37:51.815407 1680 raft_consensus.cc:2454] T a931a8da37a645439f59a12eec52c35c P 1da850e274ca47e78d4760c19a8b1998 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6dabe2324f6145ee95783fa302c6ba71 in term 0.
I20260620 18:37:51.815402 1957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a931a8da37a645439f59a12eec52c35c" candidate_uuid: "6dabe2324f6145ee95783fa302c6ba71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c03d239d0b234c5a9c19eaf8551526c5" is_pre_election: true
I20260620 18:37:51.815934 1957 raft_consensus.cc:2454] T a931a8da37a645439f59a12eec52c35c P c03d239d0b234c5a9c19eaf8551526c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6dabe2324f6145ee95783fa302c6ba71 in term 0.
I20260620 18:37:51.816102 1759 leader_election.cc:302] T a931a8da37a645439f59a12eec52c35c P 6dabe2324f6145ee95783fa302c6ba71 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1da850e274ca47e78d4760c19a8b1998, 6dabe2324f6145ee95783fa302c6ba71; no voters:
I20260620 18:37:51.816491 2029 raft_consensus.cc:2781] T a931a8da37a645439f59a12eec52c35c P 6dabe2324f6145ee95783fa302c6ba71 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:37:51.816638 2029 raft_consensus.cc:482] T a931a8da37a645439f59a12eec52c35c P 6dabe2324f6145ee95783fa302c6ba71 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:37:51.816737 2029 raft_consensus.cc:3037] T a931a8da37a645439f59a12eec52c35c P 6dabe2324f6145ee95783fa302c6ba71 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:37:51.818915 2029 raft_consensus.cc:504] T a931a8da37a645439f59a12eec52c35c P 6dabe2324f6145ee95783fa302c6ba71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dabe2324f6145ee95783fa302c6ba71" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 37791 } } peers { permanent_uuid: "1da850e274ca47e78d4760c19a8b1998" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 45681 } } peers { permanent_uuid: "c03d239d0b234c5a9c19eaf8551526c5" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 44953 } }
I20260620 18:37:51.819497 2029 leader_election.cc:288] T a931a8da37a645439f59a12eec52c35c P 6dabe2324f6145ee95783fa302c6ba71 [CANDIDATE]: Term 1 election: Requested vote from peers 1da850e274ca47e78d4760c19a8b1998 (127.22.37.1:45681), c03d239d0b234c5a9c19eaf8551526c5 (127.22.37.3:44953)
I20260620 18:37:51.819866 1680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a931a8da37a645439f59a12eec52c35c" candidate_uuid: "6dabe2324f6145ee95783fa302c6ba71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1da850e274ca47e78d4760c19a8b1998"
I20260620 18:37:51.819952 1957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a931a8da37a645439f59a12eec52c35c" candidate_uuid: "6dabe2324f6145ee95783fa302c6ba71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c03d239d0b234c5a9c19eaf8551526c5"
I20260620 18:37:51.820056 1680 raft_consensus.cc:3037] T a931a8da37a645439f59a12eec52c35c P 1da850e274ca47e78d4760c19a8b1998 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:37:51.820165 1957 raft_consensus.cc:3037] T a931a8da37a645439f59a12eec52c35c P c03d239d0b234c5a9c19eaf8551526c5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:37:51.822492 1957 raft_consensus.cc:2454] T a931a8da37a645439f59a12eec52c35c P c03d239d0b234c5a9c19eaf8551526c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6dabe2324f6145ee95783fa302c6ba71 in term 1.
I20260620 18:37:51.822775 1680 raft_consensus.cc:2454] T a931a8da37a645439f59a12eec52c35c P 1da850e274ca47e78d4760c19a8b1998 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6dabe2324f6145ee95783fa302c6ba71 in term 1.
I20260620 18:37:51.822870 1756 leader_election.cc:302] T a931a8da37a645439f59a12eec52c35c P 6dabe2324f6145ee95783fa302c6ba71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6dabe2324f6145ee95783fa302c6ba71, c03d239d0b234c5a9c19eaf8551526c5; no voters:
I20260620 18:37:51.823189 2029 raft_consensus.cc:2781] T a931a8da37a645439f59a12eec52c35c P 6dabe2324f6145ee95783fa302c6ba71 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:37:51.823616 2029 raft_consensus.cc:686] T a931a8da37a645439f59a12eec52c35c P 6dabe2324f6145ee95783fa302c6ba71 [term 1 LEADER]: Becoming Leader. State: Replica: 6dabe2324f6145ee95783fa302c6ba71, State: Running, Role: LEADER
I20260620 18:37:51.824173 2029 consensus_queue.cc:237] T a931a8da37a645439f59a12eec52c35c P 6dabe2324f6145ee95783fa302c6ba71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dabe2324f6145ee95783fa302c6ba71" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 37791 } } peers { permanent_uuid: "1da850e274ca47e78d4760c19a8b1998" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 45681 } } peers { permanent_uuid: "c03d239d0b234c5a9c19eaf8551526c5" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 44953 } }
I20260620 18:37:51.828485 1511 catalog_manager.cc:5495] T a931a8da37a645439f59a12eec52c35c P 6dabe2324f6145ee95783fa302c6ba71 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6dabe2324f6145ee95783fa302c6ba71 (127.22.37.2). New cstate: current_term: 1 leader_uuid: "6dabe2324f6145ee95783fa302c6ba71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dabe2324f6145ee95783fa302c6ba71" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 37791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1da850e274ca47e78d4760c19a8b1998" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 45681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c03d239d0b234c5a9c19eaf8551526c5" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 44953 } health_report { overall_health: UNKNOWN } } }
I20260620 18:37:51.868209 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:51.862417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57194 (local address 127.22.37.2:37791)
0620 18:37:51.862605 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:51.862612 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:51.862709 (+ 97us) server_negotiation.cc:407] Connection header received
0620 18:37:51.862949 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:51.862953 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:51.863037 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:51.863182 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:37:51.863563 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:51.864239 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:51.865754 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:51.866167 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:51.866298 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:51.866761 (+ 463us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:37:51.867216 (+ 455us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:37:51.867364 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:51.867475 (+ 111us) server_negotiation.cc:299] Negotiation successful
0620 18:37:51.867552 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 18:37:51.876134 1957 raft_consensus.cc:1261] T a931a8da37a645439f59a12eec52c35c P c03d239d0b234c5a9c19eaf8551526c5 [term 1 FOLLOWER]: Refusing update from remote peer 6dabe2324f6145ee95783fa302c6ba71: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:37:51.876135 1680 raft_consensus.cc:1261] T a931a8da37a645439f59a12eec52c35c P 1da850e274ca47e78d4760c19a8b1998 [term 1 FOLLOWER]: Refusing update from remote peer 6dabe2324f6145ee95783fa302c6ba71: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:37:51.877192 2032 consensus_queue.cc:1035] T a931a8da37a645439f59a12eec52c35c P 6dabe2324f6145ee95783fa302c6ba71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c03d239d0b234c5a9c19eaf8551526c5" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 44953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:37:51.877601 2029 consensus_queue.cc:1035] T a931a8da37a645439f59a12eec52c35c P 6dabe2324f6145ee95783fa302c6ba71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1da850e274ca47e78d4760c19a8b1998" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 45681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:37:51.889047 2035 mvcc.cc:205] Tried to move back new op lower bound from 7298992831994675200 to 7298992831795568640. Current Snapshot: MvccSnapshot[applied={T|T < 7298992831994675200}]
I20260620 18:37:51.896874 2036 mvcc.cc:205] Tried to move back new op lower bound from 7298992831994675200 to 7298992831795568640. Current Snapshot: MvccSnapshot[applied={T|T < 7298992831994675200 or (T in {7298992831994675200})}]
I20260620 18:37:51.903081 1512 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:37:51.903365 1512 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:37:51.906934 1512 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0d37b1d9f74d41029b7ef81e5012d67f: Sending DeleteTablet for 3 replicas of tablet a931a8da37a645439f59a12eec52c35c
I20260620 18:37:51.907954 1803 tablet_service.cc:1514] Processing DeleteTablet for tablet a931a8da37a645439f59a12eec52c35c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:37:51 UTC) from {username='kudu', principal='kudu/127.22.37.62@KRBTEST.COM'} at 127.0.0.1:57184
I20260620 18:37:51.908073 1937 tablet_service.cc:1514] Processing DeleteTablet for tablet a931a8da37a645439f59a12eec52c35c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:37:51 UTC) from {username='kudu', principal='kudu/127.22.37.62@KRBTEST.COM'} at 127.0.0.1:48934
I20260620 18:37:51.908077 1660 tablet_service.cc:1514] Processing DeleteTablet for tablet a931a8da37a645439f59a12eec52c35c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:37:51 UTC) from {username='kudu', principal='kudu/127.22.37.62@KRBTEST.COM'} at 127.0.0.1:50710
I20260620 18:37:51.910430 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 1592
I20260620 18:37:51.924185 2047 tablet_replica.cc:330] T a931a8da37a645439f59a12eec52c35c P 1da850e274ca47e78d4760c19a8b1998: stopping tablet replica
I20260620 18:37:51.925401 2047 raft_consensus.cc:2229] T a931a8da37a645439f59a12eec52c35c P 1da850e274ca47e78d4760c19a8b1998 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:37:51.926038 2047 raft_consensus.cc:2258] T a931a8da37a645439f59a12eec52c35c P 1da850e274ca47e78d4760c19a8b1998 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:37:51.928442 2047 ts_tablet_manager.cc:1905] T a931a8da37a645439f59a12eec52c35c P 1da850e274ca47e78d4760c19a8b1998: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:37:51.937407 2047 ts_tablet_manager.cc:1918] T a931a8da37a645439f59a12eec52c35c P 1da850e274ca47e78d4760c19a8b1998: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:37:51.937616 2047 log.cc:1196] T a931a8da37a645439f59a12eec52c35c P 1da850e274ca47e78d4760c19a8b1998: Deleting WAL directory at /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/minicluster-data/ts-0/wal/wals/a931a8da37a645439f59a12eec52c35c
I20260620 18:37:51.938161 2047 ts_tablet_manager.cc:1939] T a931a8da37a645439f59a12eec52c35c P 1da850e274ca47e78d4760c19a8b1998: Deleting consensus metadata
I20260620 18:37:51.939102 2046 tablet_replica.cc:330] T a931a8da37a645439f59a12eec52c35c P c03d239d0b234c5a9c19eaf8551526c5: stopping tablet replica
I20260620 18:37:51.940140 1499 catalog_manager.cc:4841] TS 1da850e274ca47e78d4760c19a8b1998 (127.22.37.1:45681): tablet a931a8da37a645439f59a12eec52c35c (table test-table [id=d2b755d7dadf4ad59ef704fd357b5967]) successfully deleted
I20260620 18:37:51.941954 2046 raft_consensus.cc:2229] T a931a8da37a645439f59a12eec52c35c P c03d239d0b234c5a9c19eaf8551526c5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:37:51.942468 2046 raft_consensus.cc:2258] T a931a8da37a645439f59a12eec52c35c P c03d239d0b234c5a9c19eaf8551526c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:37:51.943205 2045 tablet_replica.cc:330] T a931a8da37a645439f59a12eec52c35c P 6dabe2324f6145ee95783fa302c6ba71: stopping tablet replica
I20260620 18:37:51.943670 2045 raft_consensus.cc:2229] T a931a8da37a645439f59a12eec52c35c P 6dabe2324f6145ee95783fa302c6ba71 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:37:51.944409 2045 raft_consensus.cc:2258] T a931a8da37a645439f59a12eec52c35c P 6dabe2324f6145ee95783fa302c6ba71 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:37:51.944761 2046 ts_tablet_manager.cc:1905] T a931a8da37a645439f59a12eec52c35c P c03d239d0b234c5a9c19eaf8551526c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:37:51.946187 2045 ts_tablet_manager.cc:1905] T a931a8da37a645439f59a12eec52c35c P 6dabe2324f6145ee95783fa302c6ba71: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 20 18:37:51 dist-test-slave-5lgp krb5kdc[1036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980667, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.37.1@KRBTEST.COM
I20260620 18:37:51.952507 2046 ts_tablet_manager.cc:1918] T a931a8da37a645439f59a12eec52c35c P c03d239d0b234c5a9c19eaf8551526c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:37:51.952764 2046 log.cc:1196] T a931a8da37a645439f59a12eec52c35c P c03d239d0b234c5a9c19eaf8551526c5: Deleting WAL directory at /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/minicluster-data/ts-2/wal/wals/a931a8da37a645439f59a12eec52c35c
I20260620 18:37:51.953325 2046 ts_tablet_manager.cc:1939] T a931a8da37a645439f59a12eec52c35c P c03d239d0b234c5a9c19eaf8551526c5: Deleting consensus metadata
I20260620 18:37:51.953778 2045 ts_tablet_manager.cc:1918] T a931a8da37a645439f59a12eec52c35c P 6dabe2324f6145ee95783fa302c6ba71: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:37:51.953969 2045 log.cc:1196] T a931a8da37a645439f59a12eec52c35c P 6dabe2324f6145ee95783fa302c6ba71: Deleting WAL directory at /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980527431848-22676-0/minicluster-data/ts-1/wal/wals/a931a8da37a645439f59a12eec52c35c
I20260620 18:37:51.954546 2045 ts_tablet_manager.cc:1939] T a931a8da37a645439f59a12eec52c35c P 6dabe2324f6145ee95783fa302c6ba71: Deleting consensus metadata
I20260620 18:37:51.955350 1496 catalog_manager.cc:4841] TS c03d239d0b234c5a9c19eaf8551526c5 (127.22.37.3:44953): tablet a931a8da37a645439f59a12eec52c35c (table test-table [id=d2b755d7dadf4ad59ef704fd357b5967]) successfully deleted
I20260620 18:37:51.955669 2020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:51.912237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50722 (local address 127.22.37.1:45681)
0620 18:37:51.912443 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:51.912452 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:51.936024 (+ 23572us) server_negotiation.cc:407] Connection header received
0620 18:37:51.936578 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:51.936584 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:51.936660 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:51.936737 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:51.939449 (+ 2712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:51.940516 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:51.942342 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:51.942989 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:51.943104 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:51.948193 (+ 5089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:51.948207 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:51.948223 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:37:51.948266 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:51.952107 (+ 3841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:51.953020 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:51.953031 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:51.953038 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:51.953194 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:51.953714 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:51.953719 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:51.953722 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:51.954170 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:51.954454 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:51.955001 (+ 547us) server_negotiation.cc:299] Negotiation successful
0620 18:37:51.955263 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 18:37:51.956270 1499 catalog_manager.cc:4841] TS 6dabe2324f6145ee95783fa302c6ba71 (127.22.37.2:37791): tablet a931a8da37a645439f59a12eec52c35c (table test-table [id=d2b755d7dadf4ad59ef704fd357b5967]) successfully deleted
I20260620 18:37:51.957113 1720 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:37:52.046932 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 1592
I20260620 18:37:52.070798 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 1731
Jun 20 18:37:52 dist-test-slave-5lgp krb5kdc[1036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980667, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.37.2@KRBTEST.COM
I20260620 18:37:52.088565 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:52.071641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57208 (local address 127.22.37.2:37791)
0620 18:37:52.071840 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:52.071846 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:52.071868 (+ 22us) server_negotiation.cc:407] Connection header received
0620 18:37:52.072046 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:52.072052 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:52.072144 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:52.072265 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:52.073641 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:52.074899 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:52.076258 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:52.076927 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:52.077059 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:52.081475 (+ 4416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:52.081499 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:52.081519 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:37:52.081573 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:52.085542 (+ 3969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:52.086248 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:52.086259 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:52.086263 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:52.086457 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:52.086864 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:52.086870 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:52.086873 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:52.087353 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:52.087688 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:52.087934 (+ 246us) server_negotiation.cc:299] Negotiation successful
0620 18:37:52.088149 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 18:37:52.089385 1863 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:37:52.200793 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 1731
I20260620 18:37:52.219089 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 1874
Jun 20 18:37:52 dist-test-slave-5lgp krb5kdc[1036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980667, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.37.3@KRBTEST.COM
I20260620 18:37:52.235729 2019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:52.219895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48944 (local address 127.22.37.3:44953)
0620 18:37:52.220070 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:52.220076 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:52.220125 (+ 49us) server_negotiation.cc:407] Connection header received
0620 18:37:52.220374 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:52.220378 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:52.220439 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:52.220553 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:52.221994 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:52.222917 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:52.225008 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:52.225554 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:52.225686 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:52.229965 (+ 4279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:52.229980 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:52.229997 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:37:52.230032 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:52.232932 (+ 2900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:52.233704 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:52.233712 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:52.233714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:52.233835 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:52.234229 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:52.234236 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:52.234243 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:52.234584 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:52.234875 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:52.235179 (+ 304us) server_negotiation.cc:299] Negotiation successful
0620 18:37:52.235379 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 18:37:52.236377 1997 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:37:52.325227 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 1874
I20260620 18:37:52.343663 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 1480
I20260620 18:37:52.344805 1542 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:37:52.456039 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 1480
I20260620 18:37:52.480041 22676 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 18:37:52.815677 22676 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 18:37:52.815898 22676 mini_ranger.cc:66] Stopping Ranger...
I20260620 18:37:53.161914 22676 mini_ranger.cc:68] Stopped Ranger
2026-06-20 18:37:53.162 UTC [1066] LOG: received smart shutdown request
2026-06-20 18:37:53.165 UTC [1066] LOG: background worker "logical replication launcher" (PID 1075) exited with exit code 1
2026-06-20 18:37:53.165 UTC [1070] LOG: shutting down
2026-06-20 18:37:53.336 UTC [1066] LOG: database system is shut down
2026-06-20T18:37:53Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (48330 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:37:53 dist-test-slave-5lgp krb5kdc[2061](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:37:53 dist-test-slave-5lgp krb5kdc[2061](info): set up 2 sockets
Jun 20 18:37:53 dist-test-slave-5lgp krb5kdc[2061](info): commencing operation
krb5kdc: starting...
W20260620 18:37:54.565791 22676 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.006s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/krb5kdc/test-user.keytab.
Jun 20 18:37:54 dist-test-slave-5lgp krb5kdc[2061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980674, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:37:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:37:54Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.22.37.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.22.37.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.22.37.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/krb5kdc/rangeradmin_127.22.37.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.22.37.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/krb5kdc/rangeradmin_127.22.37.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.22.37.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.22.37.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.22.37.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/krb5kdc/rangerlookup_127.22.37.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.22.37.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/krb5kdc/rangerlookup_127.22.37.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.22.37.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.22.37.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.37.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/krb5kdc/HTTP_127.22.37.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.22.37.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/krb5kdc/HTTP_127.22.37.20@KRBTEST.COM.keytab.
I20260620 18:37:54.890278 22676 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task1UXpXs/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/postgres -l logfile start
2026-06-20 18:37:57.362 UTC [2091] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 18:37:57.362 UTC [2091] LOG: listening on IPv4 address "127.22.37.20", port 60011
2026-06-20 18:37:57.366 UTC [2091] LOG: listening on Unix socket "/tmp/.s.PGSQL.60011"
2026-06-20 18:37:57.388 UTC [2094] LOG: database system was shut down at 2026-06-20 18:37:55 UTC
2026-06-20 18:37:57.391 UTC [2091] LOG: database system is ready to accept connections
I20260620 18:37:58.365533 22676 mini_postgres.cc:96] Postgres bound to 60011
2026-06-20 18:37:58.382 UTC [2102] FATAL: database "slave" does not exist
127.22.37.20:60011 - accepting connections
I20260620 18:37:58.383138 22676 mini_ranger.cc:161] Starting Ranger...
I20260620 18:37:58.408771 22676 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 18:37:58.881577 22676 mini_ranger.cc:87] Created ranger Postgres database
I20260620 18:37:58.881672 22676 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ranger-admin
2026-06-20 18:37:59,151 [I] DB FLAVOR :POSTGRES
2026-06-20 18:37:59,152 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 18:37:59,152 [I] Checking connection..
2026-06-20 18:37:59,152 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1UXpXs/build/asan/bin/postgresql.jar:/tmp/dist-test-task1UXpXs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.37.20:60011/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 18:37:59,408 [I] Checking connection passed.
2026-06-20 18:37:59,408 [I] --------- Verifying version history table ---------
2026-06-20 18:37:59,408 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1UXpXs/build/asan/bin/postgresql.jar:/tmp/dist-test-task1UXpXs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.37.20:60011/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 18:37:59,642 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 18:37:59,642 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1UXpXs/build/asan/bin/postgresql.jar:/tmp/dist-test-task1UXpXs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.37.20:60011/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 18:37:59,876 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 18:37:59,877 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 18:37:59,877 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1UXpXs/build/asan/bin/postgresql.jar:/tmp/dist-test-task1UXpXs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.37.20:60011/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task1UXpXs/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 18:38:00.128 UTC [2166] WARNING: there is no transaction in progress
2026-06-20 18:38:00,134 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 18:38:00,134 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1UXpXs/build/asan/bin/postgresql.jar:/tmp/dist-test-task1UXpXs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.37.20:60011/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 18:38:00,374 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 18:38:00,375 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 18:38:00,375 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1UXpXs/build/asan/bin/postgresql.jar:/tmp/dist-test-task1UXpXs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.37.20:60011/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 18:38:00,612 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1UXpXs/build/asan/bin/postgresql.jar:/tmp/dist-test-task1UXpXs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.37.20:60011/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 18:38:00,858 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1UXpXs/build/asan/bin/postgresql.jar:/tmp/dist-test-task1UXpXs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.37.20:60011/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-5lgp.c.gcp-upstream.internal','N') ;"
2026-06-20 18:38:01,111 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1UXpXs/build/asan/bin/postgresql.jar:/tmp/dist-test-task1UXpXs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.37.20:60011/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 18:38:01,377 [I] Table x_portal_user does not exist in database ranger
2026-06-20 18:38:01,377 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1UXpXs/build/asan/bin/postgresql.jar:/tmp/dist-test-task1UXpXs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.37.20:60011/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 18:38:01,626 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 18:38:01,626 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1UXpXs/build/asan/bin/postgresql.jar:/tmp/dist-test-task1UXpXs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.37.20:60011/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 18:38:01,876 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1UXpXs/build/asan/bin/postgresql.jar:/tmp/dist-test-task1UXpXs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.37.20:60011/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 18:38:02,121 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 18:38:02,121 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1UXpXs/build/asan/bin/postgresql.jar:/tmp/dist-test-task1UXpXs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.37.20:60011/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task1UXpXs/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 18:38:03.052 UTC [2274] WARNING: there is no transaction in progress
2026-06-20 18:38:03.068 UTC [2274] WARNING: there is no transaction in progress
2026-06-20 18:38:03.084 UTC [2274] WARNING: there is no transaction in progress
2026-06-20 18:38:03.099 UTC [2274] WARNING: there is no transaction in progress
2026-06-20 18:38:03.117 UTC [2274] WARNING: there is no transaction in progress
2026-06-20 18:38:03.134 UTC [2274] WARNING: there is no transaction in progress
2026-06-20 18:38:03.197 UTC [2274] WARNING: there is no transaction in progress
2026-06-20 18:38:03.204 UTC [2274] WARNING: there is no transaction in progress
2026-06-20 18:38:03.218 UTC [2274] WARNING: there is no transaction in progress
2026-06-20 18:38:03.228 UTC [2274] WARNING: there is no transaction in progress
2026-06-20 18:38:03.238 UTC [2274] WARNING: there is no transaction in progress
2026-06-20 18:38:03.252 UTC [2274] WARNING: there is no transaction in progress
2026-06-20 18:38:03.261 UTC [2274] WARNING: there is no transaction in progress
2026-06-20 18:38:03.268 UTC [2274] WARNING: there is no transaction in progress
2026-06-20 18:38:03.274 UTC [2274] WARNING: there is no transaction in progress
2026-06-20 18:38:03.281 UTC [2274] WARNING: there is no transaction in progress
2026-06-20 18:38:03,902 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 18:38:03,903 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1UXpXs/build/asan/bin/postgresql.jar:/tmp/dist-test-task1UXpXs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.37.20:60011/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 18:38:04,155 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 18:38:04,155 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1UXpXs/build/asan/bin/postgresql.jar:/tmp/dist-test-task1UXpXs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.37.20:60011/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 18:38:04,402 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 18:38:04,402 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1UXpXs/build/asan/bin/postgresql.jar:/tmp/dist-test-task1UXpXs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.37.20:60011/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 18:38:04,657 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 18:38:04,658 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1UXpXs/build/asan/bin/postgresql.jar:/tmp/dist-test-task1UXpXs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.37.20:60011/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 18:38:04,901 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1UXpXs/build/asan/bin/postgresql.jar:/tmp/dist-test-task1UXpXs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.37.20:60011/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 18:38:05,133 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1UXpXs/build/asan/bin/postgresql.jar:/tmp/dist-test-task1UXpXs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.37.20:60011/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-5lgp.c.gcp-upstream.internal';"
2026-06-20 18:38:05,391 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 18:38:05,391 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1UXpXs/build/asan/bin/postgresql.jar:/tmp/dist-test-task1UXpXs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.37.20:60011/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 18:38:05,617 [I] DB_PATCHES have already been applied
I20260620 18:38:05.623863 22676 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ranger-admin:/tmp/dist-test-task1UXpXs/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task1UXpXs/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task1UXpXs/build/asan/bin/postgresql.jar:/tmp/dist-test-task1UXpXs/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 18:38:05.623951 22676 mini_ranger.cc:193] Using host: 127.22.37.20
I20260620 18:38:05.638348 22676 mini_ranger.cc:227] Ranger admin URL: http://127.22.37.20:38659
Jun 20, 2026 6:38:05 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 6:38:05 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 6:38:06 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 6:38:06 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 6:38:06 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 6:38:06 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 6:38:06 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 18:38:06.763212 22676 mini_ranger.cc:160] Time spent starting Ranger: real 8.380s user 0.000s sys 0.037s
Jun 20, 2026 6:38:26 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 6:38:26 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 20, 2026 6:38:26 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 6:38:26 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 6:38:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 6:38:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 6:38:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 6:38:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 6:38:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 6:38:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 6:38:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 6:38:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 6:38:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 6:38:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 6:38:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 6:38:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 6:38:26 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 6:38:27 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 18:38:27.762701 22676 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.22.37.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.22.37.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.22.37.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/krb5kdc/rangerkms_127.22.37.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.22.37.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/krb5kdc/rangerkms_127.22.37.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.22.37.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.22.37.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.37.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/krb5kdc/HTTP_127.22.37.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.22.37.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/krb5kdc/HTTP_127.22.37.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 18:38:27 dist-test-slave-5lgp krb5kdc[2061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980707, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 18:38:27.924218 22676 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 18:38:27.947321 22676 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 18:38:28.802560 22676 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 18:38:28.802680 22676 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ranger-kms
I20260620 18:38:28.802733 22676 mini_ranger_kms.cc:227] Using postgres at 127.22.37.20:60011
2026-06-20 18:38:29,151 [I] DB FLAVOR :POSTGRES
2026-06-20 18:38:29,152 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 18:38:29,152 [I] Checking connection
2026-06-20 18:38:29,152 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1UXpXs/build/asan/bin/postgresql.jar:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.37.20:60011/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 18:38:29,403 [I] connection success
2026-06-20 18:38:29,403 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 18:38:29,403 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 18:38:29,403 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1UXpXs/build/asan/bin/postgresql.jar:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.37.20:60011/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 18:38:29,631 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 18:38:29,631 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 18:38:29,632 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 18:38:29,632 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1UXpXs/build/asan/bin/postgresql.jar:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.37.20:60011/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 18:38:29,883 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 18:38:30.321201 22676 mini_ranger_kms.cc:310] Created kms service
I20260620 18:38:30.513448 22676 mini_ranger_kms.cc:326] Created kudu user
I20260620 18:38:30.559778 22676 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 18:38:30 dist-test-slave-5lgp krb5kdc[2061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980707, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.22.37.20@KRBTEST.COM
I20260620 18:38:30.975819 22676 mini_ranger_kms.cc:384] Added ranger policy
I20260620 18:38:30.976203 22676 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ranger-kms:/tmp/dist-test-task1UXpXs/build/asan/bin/postgresql.jar:/tmp/dist-test-task1UXpXs/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task1UXpXs/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 18:38:30.976260 22676 mini_ranger_kms.cc:240] Using host: 127.22.37.20
I20260620 18:38:30.994906 22676 mini_ranger_kms.cc:276] Ranger KMS PID: 2454
I20260620 18:38:30.995065 22676 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.22.37.20:58809
Jun 20, 2026 6:38:31 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 6:38:31 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 6:38:32 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 6:38:32 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 6:38:32 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 6:38:32 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Jun 20 18:38:33 dist-test-slave-5lgp krb5kdc[2061](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781980713, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.22.37.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:33.282049 22676 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.358s user 0.072s sys 0.289s
I20260620 18:38:33.282299 22676 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 18:38:33.282469 22676 mini_ranger_kms.cc:401] 127.22.37.20:58809/kms/v1/keys
Jun 20 18:38:33 dist-test-slave-5lgp krb5kdc[2061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980713, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.22.37.20@KRBTEST.COM for HTTP/127.22.37.20@KRBTEST.COM
Jun 20 18:38:34 dist-test-slave-5lgp krb5kdc[2061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980713, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.22.37.20@KRBTEST.COM for HTTP/127.22.37.20@KRBTEST.COM
Jun 20 18:38:36 dist-test-slave-5lgp krb5kdc[2061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980716, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.22.37.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.37.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.37.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab.
Entry for principal kudu/127.22.37.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab.
I20260620 18:38:36.329228 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.62:44233
--webserver_interface=127.22.37.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab
--principal=kudu/127.22.37.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:37441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.37.62:44233
--ranger_config_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.37.20:58809/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:38:36.660800 2505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:36.661154 2505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:36.661324 2505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:36.668557 2505 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:38:36.668727 2505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:36.668833 2505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:36.668944 2505 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:38:36.669026 2505 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:38:36.679495 2505 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:37441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/master-0/wal
--ranger_kms_url=127.22.37.20:58809/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.22.37.62:44233
--ranger_config_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab
--principal=kudu/127.22.37.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.62:44233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.37.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:38:36.680902 2505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:36.682283 2505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:36.694360 2510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:36.694411 2511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:36.694301 2513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:36.695097 2505 server_base.cc:1034] running on GCE node
I20260620 18:38:36.695986 2505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:36.697490 2505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:36.698720 2505 hybrid_clock.cc:648] HybridClock initialized: now 1781980716698682 us; error 35 us; skew 500 ppm
I20260620 18:38:36.699146 2505 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:38:36 dist-test-slave-5lgp krb5kdc[2061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980716, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:36.702490 2505 init.cc:377] Logged in from keytab as kudu/127.22.37.62@KRBTEST.COM (short username kudu)
I20260620 18:38:36.704011 2505 webserver.cc:460] Webserver started at http://127.22.37.62:44035/ using document root <none> and password file <none>
I20260620 18:38:36.704682 2505 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:36.704821 2505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:36.705132 2505 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 18:38:36 dist-test-slave-5lgp krb5kdc[2061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980716, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.62@KRBTEST.COM for HTTP/127.22.37.20@KRBTEST.COM
I20260620 18:38:36.771535 2505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/master-0/data/instance:
uuid: "6893290e7c064c4eb0cf819f9ccbe23b"
format_stamp: "Formatted at 2026-06-20 18:38:36 on dist-test-slave-5lgp"
server_key: "7f7b9c3e4029ece6c0340cc876fed347"
server_key_iv: "22b60488d229189e6f8c6709e6f1bc65"
server_key_version: "kuduclusterkey@0"
I20260620 18:38:36.772821 2505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/master-0/wal/instance:
uuid: "6893290e7c064c4eb0cf819f9ccbe23b"
format_stamp: "Formatted at 2026-06-20 18:38:36 on dist-test-slave-5lgp"
server_key: "7f7b9c3e4029ece6c0340cc876fed347"
server_key_iv: "22b60488d229189e6f8c6709e6f1bc65"
server_key_version: "kuduclusterkey@0"
I20260620 18:38:36.780730 2505 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.001s
I20260620 18:38:36.802817 2520 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:36.804795 2505 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260620 18:38:36.805006 2505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/master-0/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/master-0/wal
uuid: "6893290e7c064c4eb0cf819f9ccbe23b"
format_stamp: "Formatted at 2026-06-20 18:38:36 on dist-test-slave-5lgp"
server_key: "7f7b9c3e4029ece6c0340cc876fed347"
server_key_iv: "22b60488d229189e6f8c6709e6f1bc65"
server_key_version: "kuduclusterkey@0"
I20260620 18:38:36.805289 2505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:36.836128 2505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:36.841586 2505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:36.841980 2505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:36.862957 2505 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.62:44233
I20260620 18:38:36.862958 2571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.62:44233 every 8 connection(s)
I20260620 18:38:36.864837 2505 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
I20260620 18:38:36.865258 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 2505
I20260620 18:38:36.865597 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/master-0/wal/instance
I20260620 18:38:36.866518 22676 external_mini_cluster.cc:1428] Setting key 5551b6146a03c6ccea1e26e25cd4f96d
I20260620 18:38:36.870266 2572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:36.886718 2572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6893290e7c064c4eb0cf819f9ccbe23b: Bootstrap starting.
I20260620 18:38:36.891196 2572 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6893290e7c064c4eb0cf819f9ccbe23b: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:36.892555 2572 log.cc:824] T 00000000000000000000000000000000 P 6893290e7c064c4eb0cf819f9ccbe23b: Log is configured to *not* fsync() on all Append() calls
I20260620 18:38:36.896040 2572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6893290e7c064c4eb0cf819f9ccbe23b: No bootstrap required, opened a new log
Jun 20 18:38:36 dist-test-slave-5lgp krb5kdc[2061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980716, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM
I20260620 18:38:36.905360 2572 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6893290e7c064c4eb0cf819f9ccbe23b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6893290e7c064c4eb0cf819f9ccbe23b" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 44233 } }
I20260620 18:38:36.905901 2572 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6893290e7c064c4eb0cf819f9ccbe23b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:36.906041 2572 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6893290e7c064c4eb0cf819f9ccbe23b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6893290e7c064c4eb0cf819f9ccbe23b, State: Initialized, Role: FOLLOWER
I20260620 18:38:36.906755 2572 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6893290e7c064c4eb0cf819f9ccbe23b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6893290e7c064c4eb0cf819f9ccbe23b" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 44233 } }
I20260620 18:38:36.907022 2572 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6893290e7c064c4eb0cf819f9ccbe23b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:38:36.907214 2572 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6893290e7c064c4eb0cf819f9ccbe23b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:38:36.907426 2572 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6893290e7c064c4eb0cf819f9ccbe23b [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:36.910270 2575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:36.874505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43418 (local address 127.22.37.62:44233)
0620 18:38:36.883380 (+ 8875us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:36.883400 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:36.883466 (+ 66us) server_negotiation.cc:407] Connection header received
0620 18:38:36.886242 (+ 2776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:36.886282 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:36.886905 (+ 623us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:36.887440 (+ 535us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:36.889238 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:36.891466 (+ 2228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:36.893493 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:36.894299 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:36.894491 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:36.899788 (+ 5297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:36.899804 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:36.899826 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:36.899890 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:36.904874 (+ 4984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:36.905821 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:36.905831 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:36.905841 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:36.906052 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:36.906544 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:36.906549 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:36.906552 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:36.907325 (+ 773us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:36.907639 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:36.908207 (+ 568us) server_negotiation.cc:299] Negotiation successful
0620 18:38:36.908882 (+ 675us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8558,"thread_start_us":8310,"threads_started":1}
I20260620 18:38:36.910854 2572 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6893290e7c064c4eb0cf819f9ccbe23b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6893290e7c064c4eb0cf819f9ccbe23b" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 44233 } }
I20260620 18:38:36.911535 2572 leader_election.cc:302] T 00000000000000000000000000000000 P 6893290e7c064c4eb0cf819f9ccbe23b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6893290e7c064c4eb0cf819f9ccbe23b; no voters:
I20260620 18:38:36.912318 2572 leader_election.cc:288] T 00000000000000000000000000000000 P 6893290e7c064c4eb0cf819f9ccbe23b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:38:36.912493 2577 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6893290e7c064c4eb0cf819f9ccbe23b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:38:36.916181 2572 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6893290e7c064c4eb0cf819f9ccbe23b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:38:36.919560 2577 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6893290e7c064c4eb0cf819f9ccbe23b [term 1 LEADER]: Becoming Leader. State: Replica: 6893290e7c064c4eb0cf819f9ccbe23b, State: Running, Role: LEADER
I20260620 18:38:36.920275 2577 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6893290e7c064c4eb0cf819f9ccbe23b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6893290e7c064c4eb0cf819f9ccbe23b" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 44233 } }
I20260620 18:38:36.927026 2572 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 18:38:36.927541 2578 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6893290e7c064c4eb0cf819f9ccbe23b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6893290e7c064c4eb0cf819f9ccbe23b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6893290e7c064c4eb0cf819f9ccbe23b" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 44233 } } }
I20260620 18:38:36.927904 2578 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6893290e7c064c4eb0cf819f9ccbe23b [sys.catalog]: This master's current role is: LEADER
I20260620 18:38:36.931525 2579 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6893290e7c064c4eb0cf819f9ccbe23b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6893290e7c064c4eb0cf819f9ccbe23b. Latest consensus state: current_term: 1 leader_uuid: "6893290e7c064c4eb0cf819f9ccbe23b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6893290e7c064c4eb0cf819f9ccbe23b" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 44233 } } }
I20260620 18:38:36.932288 2579 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6893290e7c064c4eb0cf819f9ccbe23b [sys.catalog]: This master's current role is: LEADER
I20260620 18:38:36.942579 2592 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:38:36.948776 2592 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:38:36.958949 2592 catalog_manager.cc:1347] Generated new cluster ID: 3ff51d1d4bde45ad986e8fc0b9c14610
I20260620 18:38:36.959057 2592 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:38:36.968730 2592 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:38:36.970709 2592 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:38:36.978286 2592 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6893290e7c064c4eb0cf819f9ccbe23b: Generated new TSK 0
I20260620 18:38:36.979316 2592 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.37.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.37.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.37.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.1.keytab.
Entry for principal kudu/127.22.37.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.1.keytab.
I20260620 18:38:37.744801 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.37.20:58809/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.37.1:0
--local_ip_for_outbound_sockets=127.22.37.1
--webserver_interface=127.22.37.1
--webserver_port=0
--tserver_master_addrs=127.22.37.62:44233
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.1.keytab
--principal=kudu/127.22.37.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:37441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 18:38:37 dist-test-slave-5lgp krb5kdc[2061](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781980717, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 18:38:38.171741 2617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:38.172145 2617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:38.172374 2617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:38.184404 2617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:38.184633 2617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:38.184859 2617 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.37.1
I20260620 18:38:38.201155 2617 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:37441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.22.37.20:58809/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.1.keytab
--principal=kudu/127.22.37.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.37.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.37.1
--webserver_port=0
--tserver_master_addrs=127.22.37.62:44233
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.37.1
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:38:38.202989 2617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:38.204820 2617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:38.224037 2630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:38.224031 2628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:38.224417 2627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:38.225950 2617 server_base.cc:1034] running on GCE node
I20260620 18:38:38.226667 2617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:38.227720 2617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:38.228938 2617 hybrid_clock.cc:648] HybridClock initialized: now 1781980718228882 us; error 54 us; skew 500 ppm
I20260620 18:38:38.229264 2617 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:38:38 dist-test-slave-5lgp krb5kdc[2061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980718, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:38.232720 2617 init.cc:377] Logged in from keytab as kudu/127.22.37.1@KRBTEST.COM (short username kudu)
I20260620 18:38:38.234416 2617 webserver.cc:460] Webserver started at http://127.22.37.1:34617/ using document root <none> and password file <none>
I20260620 18:38:38.235145 2617 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:38.235283 2617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:38.235641 2617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 18:38:38 dist-test-slave-5lgp krb5kdc[2061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980718, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.1@KRBTEST.COM for HTTP/127.22.37.20@KRBTEST.COM
I20260620 18:38:38.262686 2617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ts-0/data/instance:
uuid: "c3c2ded58c2548c8a986e4af5cfee1d4"
format_stamp: "Formatted at 2026-06-20 18:38:38 on dist-test-slave-5lgp"
server_key: "e4eef7330bd46f615f4b0a1d2024bb50"
server_key_iv: "ec16bc619259672ae7babba304cc2063"
server_key_version: "kuduclusterkey@0"
I20260620 18:38:38.263876 2617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ts-0/wal/instance:
uuid: "c3c2ded58c2548c8a986e4af5cfee1d4"
format_stamp: "Formatted at 2026-06-20 18:38:38 on dist-test-slave-5lgp"
server_key: "e4eef7330bd46f615f4b0a1d2024bb50"
server_key_iv: "ec16bc619259672ae7babba304cc2063"
server_key_version: "kuduclusterkey@0"
I20260620 18:38:38.272840 2617 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.000s
I20260620 18:38:38.310293 2637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:38.312784 2617 fs_manager.cc:725] Time spent opening block manager: real 0.022s user 0.007s sys 0.008s
I20260620 18:38:38.313091 2617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ts-0/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ts-0/wal
uuid: "c3c2ded58c2548c8a986e4af5cfee1d4"
format_stamp: "Formatted at 2026-06-20 18:38:38 on dist-test-slave-5lgp"
server_key: "e4eef7330bd46f615f4b0a1d2024bb50"
server_key_iv: "ec16bc619259672ae7babba304cc2063"
server_key_version: "kuduclusterkey@0"
I20260620 18:38:38.313521 2617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:38.349323 2617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:38.354810 2617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:38.355218 2617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:38.357973 2617 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:38:38.366078 2617 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:38:38.366179 2617 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:38.366309 2617 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:38:38.366371 2617 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 18:38:38 dist-test-slave-5lgp krb5kdc[2061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980717, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.62@KRBTEST.COM for HTTP/127.22.37.20@KRBTEST.COM
I20260620 18:38:38.534579 2617 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.1:41815
I20260620 18:38:38.534638 2749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.1:41815 every 8 connection(s)
I20260620 18:38:38.536185 2617 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ts-0/data/info.pb
I20260620 18:38:38.546550 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 2617
I20260620 18:38:38.546795 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ts-0/wal/instance
I20260620 18:38:38.547469 22676 external_mini_cluster.cc:1428] Setting key cec4dd1921fe454b756120370a0e917a
Jun 20 18:38:38 dist-test-slave-5lgp krb5kdc[2061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980718, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.1@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM
I20260620 18:38:38.572340 2753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:38.540511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.1:44577 (local address 127.22.37.62:44233)
0620 18:38:38.553861 (+ 13350us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:38.553871 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:38.555070 (+ 1199us) server_negotiation.cc:407] Connection header received
0620 18:38:38.557451 (+ 2381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:38.557457 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:38.557573 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:38.557717 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:38.559207 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:38.559972 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:38.561584 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:38.562007 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:38.562140 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:38.565831 (+ 3691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:38.565837 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:38.565840 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:38.565872 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:38.568882 (+ 3010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:38.569751 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:38.569758 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:38.569760 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:38.569904 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:38.570386 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:38.570397 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:38.570399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:38.570672 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:38.570841 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:38.571657 (+ 816us) server_negotiation.cc:299] Negotiation successful
0620 18:38:38.571900 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13195,"thread_start_us":13082,"threads_started":1}
I20260620 18:38:38.573285 2752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:38.539881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:44233 (local address 127.22.37.1:44577)
0620 18:38:38.554499 (+ 14618us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:38.554564 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:38.557135 (+ 2571us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:38.557981 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:38.557991 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:38.558457 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:38.559043 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:38.559059 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:38.560123 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:38.560128 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:38.561456 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:38.561468 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:38.562236 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:38.562245 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:38.562493 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:38.563517 (+ 1024us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:38.563539 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:38.565616 (+ 2077us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:38.569048 (+ 3432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:38.569058 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:38.569077 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:38.569614 (+ 537us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:38.570019 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:38.570025 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:38.570029 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:38.570256 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:38.570805 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:38.570814 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:38.571181 (+ 367us) client_negotiation.cc:769] Sending connection context
0620 18:38:38.571530 (+ 349us) client_negotiation.cc:240] Negotiation successful
0620 18:38:38.572096 (+ 566us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14263,"thread_start_us":14032,"threads_started":1}
I20260620 18:38:38.575086 2750 heartbeater.cc:347] Connected to a master server at 127.22.37.62:44233
I20260620 18:38:38.575426 2750 heartbeater.cc:464] Registering TS with master...
I20260620 18:38:38.576191 2750 heartbeater.cc:511] Master 127.22.37.62:44233 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.22.37.2@KRBTEST.COM; defaulting to no policy
I20260620 18:38:38.578554 2537 ts_manager.cc:194] Registered new tserver with Master: c3c2ded58c2548c8a986e4af5cfee1d4 (127.22.37.1:41815)
I20260620 18:38:38.580300 2537 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.37.1@KRBTEST.COM'} at 127.22.37.1:44577
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.37.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.37.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.2.keytab.
Entry for principal kudu/127.22.37.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.2.keytab.
I20260620 18:38:38.614245 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.37.20:58809/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.37.2:0
--local_ip_for_outbound_sockets=127.22.37.2
--webserver_interface=127.22.37.2
--webserver_port=0
--tserver_master_addrs=127.22.37.62:44233
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.2.keytab
--principal=kudu/127.22.37.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:37441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 18:38:38 dist-test-slave-5lgp krb5kdc[2061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980717, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.62@KRBTEST.COM for HTTP/127.22.37.20@KRBTEST.COM
W20260620 18:38:39.084988 2756 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:39.085611 2756 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:39.085772 2756 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:39.094983 2756 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:39.095105 2756 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:39.095275 2756 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.37.2
I20260620 18:38:39.105520 2756 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:37441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.22.37.20:58809/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.2.keytab
--principal=kudu/127.22.37.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.37.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.37.2
--webserver_port=0
--tserver_master_addrs=127.22.37.62:44233
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.37.2
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:38:39.106916 2756 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:39.108328 2756 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:39.123525 2771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:39.123615 2773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:39.124405 2770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:39.125682 2756 server_base.cc:1034] running on GCE node
I20260620 18:38:39.126437 2756 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:39.127637 2756 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:39.128904 2756 hybrid_clock.cc:648] HybridClock initialized: now 1781980719128818 us; error 55 us; skew 500 ppm
I20260620 18:38:39.129285 2756 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:38:39 dist-test-slave-5lgp krb5kdc[2061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980719, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:39.132510 2756 init.cc:377] Logged in from keytab as kudu/127.22.37.2@KRBTEST.COM (short username kudu)
I20260620 18:38:39.134209 2756 webserver.cc:460] Webserver started at http://127.22.37.2:46767/ using document root <none> and password file <none>
I20260620 18:38:39.134928 2756 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:39.135056 2756 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:39.135380 2756 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 18:38:39 dist-test-slave-5lgp krb5kdc[2061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980719, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.2@KRBTEST.COM for HTTP/127.22.37.20@KRBTEST.COM
I20260620 18:38:39.160843 2756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ts-1/data/instance:
uuid: "407863940edc4c2c86cc1c73bb877fbb"
format_stamp: "Formatted at 2026-06-20 18:38:39 on dist-test-slave-5lgp"
server_key: "c8bbe857e1f923a5e9ec31a92d688473"
server_key_iv: "0ad05ce85afd1fe49aeb042e0ff7250f"
server_key_version: "kuduclusterkey@0"
I20260620 18:38:39.162416 2756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ts-1/wal/instance:
uuid: "407863940edc4c2c86cc1c73bb877fbb"
format_stamp: "Formatted at 2026-06-20 18:38:39 on dist-test-slave-5lgp"
server_key: "c8bbe857e1f923a5e9ec31a92d688473"
server_key_iv: "0ad05ce85afd1fe49aeb042e0ff7250f"
server_key_version: "kuduclusterkey@0"
I20260620 18:38:39.169852 2756 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260620 18:38:39.186440 2780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:39.188087 2756 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260620 18:38:39.188277 2756 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ts-1/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ts-1/wal
uuid: "407863940edc4c2c86cc1c73bb877fbb"
format_stamp: "Formatted at 2026-06-20 18:38:39 on dist-test-slave-5lgp"
server_key: "c8bbe857e1f923a5e9ec31a92d688473"
server_key_iv: "0ad05ce85afd1fe49aeb042e0ff7250f"
server_key_version: "kuduclusterkey@0"
I20260620 18:38:39.188503 2756 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:39.206200 2756 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:39.210319 2756 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:39.210731 2756 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:39.211980 2756 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:38:39.213693 2756 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:38:39.213828 2756 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:39.213985 2756 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:38:39.214061 2756 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:39.254717 2756 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.2:39873
I20260620 18:38:39.254720 2892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.2:39873 every 8 connection(s)
I20260620 18:38:39.256152 2756 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ts-1/data/info.pb
I20260620 18:38:39.265633 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 2756
I20260620 18:38:39.265834 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ts-1/wal/instance
I20260620 18:38:39.266338 22676 external_mini_cluster.cc:1428] Setting key e291c27dcbd3098fc3c61b830742ae59
Jun 20 18:38:39 dist-test-slave-5lgp krb5kdc[2061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980719, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.2@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM
I20260620 18:38:39.278400 2895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:39.259128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.2:34435 (local address 127.22.37.62:44233)
0620 18:38:39.259621 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:39.259627 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:39.260752 (+ 1125us) server_negotiation.cc:407] Connection header received
0620 18:38:39.262704 (+ 1952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:39.262710 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:39.262840 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:39.262976 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:39.264393 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:39.265590 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:39.267848 (+ 2258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:39.268473 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:39.268615 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:39.272216 (+ 3601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:39.272228 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:39.272232 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:39.272266 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:39.275123 (+ 2857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:39.275887 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:39.275895 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:39.275899 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:39.276043 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:39.276495 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:39.276520 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:39.276524 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:39.276830 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:39.276986 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:39.277733 (+ 747us) server_negotiation.cc:299] Negotiation successful
0620 18:38:39.277963 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":239,"threads_started":1}
I20260620 18:38:39.279464 2896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:39.259683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:44233 (local address 127.22.37.2:34435)
0620 18:38:39.260577 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:39.260624 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:39.262432 (+ 1808us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:39.263205 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:39.263213 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:39.263652 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:39.264219 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:39.264236 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:39.265766 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:39.265772 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:39.267689 (+ 1917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:39.267714 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:39.268696 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:39.268705 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:39.268962 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:39.269741 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:39.269764 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:39.271990 (+ 2226us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:39.275275 (+ 3285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:39.275285 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:39.275302 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:39.275750 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:39.276154 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:39.276161 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:39.276165 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:39.276380 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:39.276966 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:39.276974 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:39.277283 (+ 309us) client_negotiation.cc:769] Sending connection context
0620 18:38:39.277634 (+ 351us) client_negotiation.cc:240] Negotiation successful
0620 18:38:39.278147 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":389,"threads_started":1}
I20260620 18:38:39.280992 2893 heartbeater.cc:347] Connected to a master server at 127.22.37.62:44233
I20260620 18:38:39.281361 2893 heartbeater.cc:464] Registering TS with master...
I20260620 18:38:39.282126 2893 heartbeater.cc:511] Master 127.22.37.62:44233 requested a full tablet report, sending...
I20260620 18:38:39.283712 2537 ts_manager.cc:194] Registered new tserver with Master: 407863940edc4c2c86cc1c73bb877fbb (127.22.37.2:39873)
I20260620 18:38:39.284571 2537 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.37.2@KRBTEST.COM'} at 127.22.37.2:34435
WARNING: no policy specified for kudu/127.22.37.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.37.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.37.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.3.keytab.
Entry for principal kudu/127.22.37.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.3.keytab.
I20260620 18:38:39.326112 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.37.20:58809/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.37.3:0
--local_ip_for_outbound_sockets=127.22.37.3
--webserver_interface=127.22.37.3
--webserver_port=0
--tserver_master_addrs=127.22.37.62:44233
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.3.keytab
--principal=kudu/127.22.37.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:37441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:38:39.583575 2750 heartbeater.cc:503] Master 127.22.37.62:44233 was elected leader, sending a full tablet report...
W20260620 18:38:39.683539 2899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:39.683841 2899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:39.684003 2899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:39.691120 2899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:39.691255 2899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:39.691437 2899 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.37.3
I20260620 18:38:39.700850 2899 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:37441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.22.37.20:58809/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.3.keytab
--principal=kudu/127.22.37.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.37.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.37.3
--webserver_port=0
--tserver_master_addrs=127.22.37.62:44233
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.37.3
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:38:39.702143 2899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:39.703410 2899 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:39.716341 2905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:39.716424 2904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:39.716374 2907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:39.717206 2899 server_base.cc:1034] running on GCE node
I20260620 18:38:39.717824 2899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:39.718674 2899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:39.719882 2899 hybrid_clock.cc:648] HybridClock initialized: now 1781980719719817 us; error 41 us; skew 500 ppm
I20260620 18:38:39.720214 2899 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:38:39 dist-test-slave-5lgp krb5kdc[2061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980719, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:39.723134 2899 init.cc:377] Logged in from keytab as kudu/127.22.37.3@KRBTEST.COM (short username kudu)
I20260620 18:38:39.724476 2899 webserver.cc:460] Webserver started at http://127.22.37.3:38561/ using document root <none> and password file <none>
I20260620 18:38:39.725104 2899 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:39.725235 2899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:39.725567 2899 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 18:38:39 dist-test-slave-5lgp krb5kdc[2061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980719, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.3@KRBTEST.COM for HTTP/127.22.37.20@KRBTEST.COM
I20260620 18:38:39.739158 2899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ts-2/data/instance:
uuid: "8dba65949de942e3ac17753e65a8aa19"
format_stamp: "Formatted at 2026-06-20 18:38:39 on dist-test-slave-5lgp"
server_key: "9acf77498e46438c691ac9cfac5e97cd"
server_key_iv: "6b7503ede98bbf113eadbe5f0288f148"
server_key_version: "kuduclusterkey@0"
I20260620 18:38:39.740176 2899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ts-2/wal/instance:
uuid: "8dba65949de942e3ac17753e65a8aa19"
format_stamp: "Formatted at 2026-06-20 18:38:39 on dist-test-slave-5lgp"
server_key: "9acf77498e46438c691ac9cfac5e97cd"
server_key_iv: "6b7503ede98bbf113eadbe5f0288f148"
server_key_version: "kuduclusterkey@0"
I20260620 18:38:39.745730 2899 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:38:39.761507 2914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:39.762924 2899 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 18:38:39.763091 2899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ts-2/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ts-2/wal
uuid: "8dba65949de942e3ac17753e65a8aa19"
format_stamp: "Formatted at 2026-06-20 18:38:39 on dist-test-slave-5lgp"
server_key: "9acf77498e46438c691ac9cfac5e97cd"
server_key_iv: "6b7503ede98bbf113eadbe5f0288f148"
server_key_version: "kuduclusterkey@0"
I20260620 18:38:39.763316 2899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:39.778170 2899 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:39.782040 2899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:39.782322 2899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:39.783169 2899 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:38:39.784479 2899 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:38:39.784576 2899 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:39.784711 2899 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:38:39.784777 2899 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:39.816708 2899 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.3:35693
I20260620 18:38:39.816743 3026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.3:35693 every 8 connection(s)
I20260620 18:38:39.818071 2899 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ts-2/data/info.pb
I20260620 18:38:39.821010 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 2899
I20260620 18:38:39.821204 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ts-2/wal/instance
I20260620 18:38:39.821762 22676 external_mini_cluster.cc:1428] Setting key b0e55d63a46c69a64330e3e58674bde7
Jun 20 18:38:39 dist-test-slave-5lgp krb5kdc[2061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980719, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.3@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM
I20260620 18:38:39.837404 3029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:39.820588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.3:43233 (local address 127.22.37.62:44233)
0620 18:38:39.821075 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:39.821081 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:39.822280 (+ 1199us) server_negotiation.cc:407] Connection header received
0620 18:38:39.824268 (+ 1988us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:39.824274 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:39.824382 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:39.824498 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:39.825886 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:39.826589 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:39.828063 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:39.828502 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:39.828598 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:39.831848 (+ 3250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:39.831853 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:39.831856 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:39.831898 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:39.834374 (+ 2476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:39.835074 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:39.835078 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:39.835080 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:39.835198 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:39.835614 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:39.835619 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:39.835627 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:39.835939 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:39.836100 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:39.836806 (+ 706us) server_negotiation.cc:299] Negotiation successful
0620 18:38:39.837022 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":232,"threads_started":1}
I20260620 18:38:39.838351 3030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:39.821274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:44233 (local address 127.22.37.3:43233)
0620 18:38:39.822131 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:39.822175 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:39.824027 (+ 1852us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:39.824735 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:39.824743 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:39.825137 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:39.825736 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:39.825753 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:39.826713 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:39.826717 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:39.827931 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:39.827952 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:39.828673 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:39.828678 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:39.828874 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:39.829555 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:39.829576 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:39.831632 (+ 2056us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:39.834487 (+ 2855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:39.834493 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:39.834505 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:39.834967 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:39.835311 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:39.835314 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:39.835316 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:39.835494 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:39.836066 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:39.836085 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:39.836462 (+ 377us) client_negotiation.cc:769] Sending connection context
0620 18:38:39.836708 (+ 246us) client_negotiation.cc:240] Negotiation successful
0620 18:38:39.837188 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":356,"threads_started":1}
I20260620 18:38:39.839520 3027 heartbeater.cc:347] Connected to a master server at 127.22.37.62:44233
I20260620 18:38:39.839812 3027 heartbeater.cc:464] Registering TS with master...
I20260620 18:38:39.840543 3027 heartbeater.cc:511] Master 127.22.37.62:44233 requested a full tablet report, sending...
I20260620 18:38:39.841930 2537 ts_manager.cc:194] Registered new tserver with Master: 8dba65949de942e3ac17753e65a8aa19 (127.22.37.3:35693)
I20260620 18:38:39.842686 2537 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.37.3@KRBTEST.COM'} at 127.22.37.3:43233
I20260620 18:38:39.849191 22676 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:38:39.850922 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 2617
Jun 20 18:38:39 dist-test-slave-5lgp krb5kdc[2061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980716, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.37.1@KRBTEST.COM
I20260620 18:38:39.868534 3032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:39.851849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56646 (local address 127.22.37.1:41815)
0620 18:38:39.852481 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:39.852488 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:39.852512 (+ 24us) server_negotiation.cc:407] Connection header received
0620 18:38:39.852566 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:39.852572 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:39.852756 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:39.852888 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:39.854505 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:39.855549 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:39.857095 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:39.857827 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:39.857929 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:39.862012 (+ 4083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:39.862023 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:39.862036 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:39.862073 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:39.865667 (+ 3594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:39.866335 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:39.866344 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:39.866348 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:39.866510 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:39.866860 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:39.866865 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:39.866868 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:39.867265 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:39.867584 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:39.867901 (+ 317us) server_negotiation.cc:299] Negotiation successful
0620 18:38:39.868159 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":359,"threads_started":1}
I20260620 18:38:39.869925 2745 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:38:39.950992 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 2617
I20260620 18:38:39.966920 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 2756
Jun 20 18:38:39 dist-test-slave-5lgp krb5kdc[2061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980716, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.37.2@KRBTEST.COM
I20260620 18:38:39.982897 3034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:39.967867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55204 (local address 127.22.37.2:39873)
0620 18:38:39.968386 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:39.968393 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:39.968420 (+ 27us) server_negotiation.cc:407] Connection header received
0620 18:38:39.968484 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:39.968490 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:39.968707 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:39.968846 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:39.970134 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:39.971146 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:39.972468 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:39.973015 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:39.973118 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:39.977340 (+ 4222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:39.977354 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:39.977414 (+ 60us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:39.977459 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:39.980202 (+ 2743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:39.980936 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:39.980943 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:39.980945 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:39.981086 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:39.981482 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:39.981485 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:39.981487 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:39.981805 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:39.982032 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:39.982297 (+ 265us) server_negotiation.cc:299] Negotiation successful
0620 18:38:39.982497 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":287,"threads_started":1}
I20260620 18:38:39.984109 2888 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:38:40.065057 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 2756
I20260620 18:38:40.080354 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 2899
Jun 20 18:38:40 dist-test-slave-5lgp krb5kdc[2061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980716, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.37.3@KRBTEST.COM
I20260620 18:38:40.097486 3036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:40.081457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48506 (local address 127.22.37.3:35693)
0620 18:38:40.082090 (+ 633us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:40.082098 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:40.082127 (+ 29us) server_negotiation.cc:407] Connection header received
0620 18:38:40.082187 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:40.082192 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:40.082415 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:40.082584 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:40.083882 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:40.084948 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:40.086496 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:40.087066 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:40.087203 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:40.091545 (+ 4342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:40.091558 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:40.091575 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:40.091615 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:40.094517 (+ 2902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:40.095309 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:40.095317 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:40.095320 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:40.095446 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:40.095827 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:40.095832 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:40.095834 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:40.096171 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:40.096494 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:40.096809 (+ 315us) server_negotiation.cc:299] Negotiation successful
0620 18:38:40.097087 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":324,"threads_started":1}
I20260620 18:38:40.098750 3022 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:38:40.181027 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 2899
I20260620 18:38:40.197007 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 2505
I20260620 18:38:40.198365 2567 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:38:40.303007 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 2505
I20260620 18:38:40.325467 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.62:44233
--webserver_interface=127.22.37.62
--webserver_port=44035
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab
--principal=kudu/127.22.37.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:37441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.37.62:44233
--ranger_config_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.37.20:58809/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.22.37.20:58809/kms with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:38:40.645941 3039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:40.646281 3039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:40.646436 3039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:40.653568 3039 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:38:40.653715 3039 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:40.653818 3039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:40.653916 3039 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:38:40.653999 3039 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:38:40.664098 3039 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:37441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.22.37.20:58809/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.22.37.62:44233
--ranger_config_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab
--principal=kudu/127.22.37.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.62:44233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.37.62
--webserver_port=44035
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:38:40.665504 3039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:40.666831 3039 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:40.678898 3047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:40.678938 3044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:40.678980 3045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:40.679773 3039 server_base.cc:1034] running on GCE node
I20260620 18:38:40.680552 3039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:40.682268 3039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:40.683502 3039 hybrid_clock.cc:648] HybridClock initialized: now 1781980720683471 us; error 62 us; skew 500 ppm
I20260620 18:38:40.683851 3039 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:38:40 dist-test-slave-5lgp krb5kdc[2061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980720, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:40.687083 3039 init.cc:377] Logged in from keytab as kudu/127.22.37.62@KRBTEST.COM (short username kudu)
I20260620 18:38:40.688635 3039 webserver.cc:460] Webserver started at http://127.22.37.62:44035/ using document root <none> and password file <none>
I20260620 18:38:40.689302 3039 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:40.689467 3039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 18:38:40 dist-test-slave-5lgp krb5kdc[2061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980720, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.62@KRBTEST.COM for HTTP/127.22.37.20@KRBTEST.COM
I20260620 18:38:40.737581 3039 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 18:38:40.740902 3055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:40.742645 3039 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 18:38:40.742852 3039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/master-0/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/master-0/wal
uuid: "6893290e7c064c4eb0cf819f9ccbe23b"
format_stamp: "Formatted at 2026-06-20 18:38:36 on dist-test-slave-5lgp"
server_key: "7f7b9c3e4029ece6c0340cc876fed347"
server_key_iv: "22b60488d229189e6f8c6709e6f1bc65"
server_key_version: "kuduclusterkey@0"
I20260620 18:38:40.743851 3039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:40.760499 3039 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:40.764657 3039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:40.764961 3039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:40.784034 3039 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.62:44233
I20260620 18:38:40.784070 3106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.62:44233 every 8 connection(s)
I20260620 18:38:40.785591 3039 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
I20260620 18:38:40.787302 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 3039
I20260620 18:38:40.788612 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.37.20:58809/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.37.1:41815
--local_ip_for_outbound_sockets=127.22.37.1
--tserver_master_addrs=127.22.37.62:44233
--webserver_port=34617
--webserver_interface=127.22.37.1
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.1.keytab
--principal=kudu/127.22.37.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:37441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.22.37.20:58809/kms with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:38:40.795604 3107 sys_catalog.cc:263] Verifying existing consensus state
I20260620 18:38:40.800504 3107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6893290e7c064c4eb0cf819f9ccbe23b: Bootstrap starting.
I20260620 18:38:40.826057 3107 log.cc:824] T 00000000000000000000000000000000 P 6893290e7c064c4eb0cf819f9ccbe23b: Log is configured to *not* fsync() on all Append() calls
I20260620 18:38:40.835397 3107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6893290e7c064c4eb0cf819f9ccbe23b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 18:38:40.835958 3107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6893290e7c064c4eb0cf819f9ccbe23b: Bootstrap complete.
I20260620 18:38:40.845232 3107 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6893290e7c064c4eb0cf819f9ccbe23b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6893290e7c064c4eb0cf819f9ccbe23b" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 44233 } }
I20260620 18:38:40.846196 3107 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6893290e7c064c4eb0cf819f9ccbe23b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6893290e7c064c4eb0cf819f9ccbe23b, State: Initialized, Role: FOLLOWER
I20260620 18:38:40.846889 3107 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6893290e7c064c4eb0cf819f9ccbe23b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6893290e7c064c4eb0cf819f9ccbe23b" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 44233 } }
I20260620 18:38:40.847122 3107 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6893290e7c064c4eb0cf819f9ccbe23b [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:38:40.847232 3107 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6893290e7c064c4eb0cf819f9ccbe23b [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:38:40.847456 3107 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6893290e7c064c4eb0cf819f9ccbe23b [term 1 FOLLOWER]: Advancing to term 2
I20260620 18:38:40.851516 3107 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6893290e7c064c4eb0cf819f9ccbe23b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6893290e7c064c4eb0cf819f9ccbe23b" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 44233 } }
I20260620 18:38:40.852025 3107 leader_election.cc:302] T 00000000000000000000000000000000 P 6893290e7c064c4eb0cf819f9ccbe23b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6893290e7c064c4eb0cf819f9ccbe23b; no voters:
I20260620 18:38:40.852669 3107 leader_election.cc:288] T 00000000000000000000000000000000 P 6893290e7c064c4eb0cf819f9ccbe23b [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 18:38:40.852805 3111 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6893290e7c064c4eb0cf819f9ccbe23b [term 2 FOLLOWER]: Leader election won for term 2
I20260620 18:38:40.855119 3107 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6893290e7c064c4eb0cf819f9ccbe23b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:38:40.859867 3111 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6893290e7c064c4eb0cf819f9ccbe23b [term 2 LEADER]: Becoming Leader. State: Replica: 6893290e7c064c4eb0cf819f9ccbe23b, State: Running, Role: LEADER
I20260620 18:38:40.860623 3111 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6893290e7c064c4eb0cf819f9ccbe23b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6893290e7c064c4eb0cf819f9ccbe23b" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 44233 } }
I20260620 18:38:40.863370 3112 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6893290e7c064c4eb0cf819f9ccbe23b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "6893290e7c064c4eb0cf819f9ccbe23b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6893290e7c064c4eb0cf819f9ccbe23b" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 44233 } } }
I20260620 18:38:40.863829 3112 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6893290e7c064c4eb0cf819f9ccbe23b [sys.catalog]: This master's current role is: LEADER
I20260620 18:38:40.864447 3113 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6893290e7c064c4eb0cf819f9ccbe23b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6893290e7c064c4eb0cf819f9ccbe23b. Latest consensus state: current_term: 2 leader_uuid: "6893290e7c064c4eb0cf819f9ccbe23b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6893290e7c064c4eb0cf819f9ccbe23b" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 44233 } } }
I20260620 18:38:40.864748 3113 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6893290e7c064c4eb0cf819f9ccbe23b [sys.catalog]: This master's current role is: LEADER
I20260620 18:38:40.866888 3123 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:38:40.887291 3123 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:38:40.889277 3123 catalog_manager.cc:1259] Loaded cluster ID: 3ff51d1d4bde45ad986e8fc0b9c14610
I20260620 18:38:40.889524 3123 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:38:40.894876 3123 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:38:40.897507 3123 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6893290e7c064c4eb0cf819f9ccbe23b: Loaded TSK: 0
I20260620 18:38:40.899786 3123 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:38:40.902674 3107 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260620 18:38:41.254094 3109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:41.254573 3109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:41.254808 3109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:41.267215 3109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:41.267436 3109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:41.267688 3109 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.37.1
I20260620 18:38:41.283685 3109 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:37441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.22.37.20:58809/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.1.keytab
--principal=kudu/127.22.37.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.37.1:41815
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.37.1
--webserver_port=34617
--tserver_master_addrs=127.22.37.62:44233
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.37.1
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:38:41.285646 3109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:41.287533 3109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:41.337682 3145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:41.340809 3146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:41.349607 3109 server_base.cc:1034] running on GCE node
W20260620 18:38:41.350121 3148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:41.351089 3109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:41.352527 3109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:41.353842 3109 hybrid_clock.cc:648] HybridClock initialized: now 1781980721353764 us; error 59 us; skew 500 ppm
I20260620 18:38:41.354329 3109 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:38:41 dist-test-slave-5lgp krb5kdc[2061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980721, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:41.362697 3109 init.cc:377] Logged in from keytab as kudu/127.22.37.1@KRBTEST.COM (short username kudu)
I20260620 18:38:41.393837 3109 webserver.cc:460] Webserver started at http://127.22.37.1:34617/ using document root <none> and password file <none>
I20260620 18:38:41.394810 3109 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:41.394992 3109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 18:38:41 dist-test-slave-5lgp krb5kdc[2061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980721, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.1@KRBTEST.COM for HTTP/127.22.37.20@KRBTEST.COM
I20260620 18:38:41.441852 3109 fs_manager.cc:709] Time spent opening directory manager: real 0.010s user 0.004s sys 0.000s
I20260620 18:38:41.474171 3157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:41.476930 3109 fs_manager.cc:725] Time spent opening block manager: real 0.033s user 0.019s sys 0.011s
I20260620 18:38:41.477159 3109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ts-0/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ts-0/wal
uuid: "c3c2ded58c2548c8a986e4af5cfee1d4"
format_stamp: "Formatted at 2026-06-20 18:38:38 on dist-test-slave-5lgp"
server_key: "e4eef7330bd46f615f4b0a1d2024bb50"
server_key_iv: "ec16bc619259672ae7babba304cc2063"
server_key_version: "kuduclusterkey@0"
I20260620 18:38:41.478714 3109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:41.513242 3109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:41.517685 3109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:41.517973 3109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:41.521996 3109 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:38:41.523597 3109 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:38:41.523718 3109 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:41.524394 3109 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:38:41.524498 3109 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:41.625721 3109 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.1:41815
I20260620 18:38:41.625782 3269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.1:41815 every 8 connection(s)
I20260620 18:38:41.627705 3109 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ts-0/data/info.pb
I20260620 18:38:41.634256 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 3109
I20260620 18:38:41.635007 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.37.20:58809/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.37.2:39873
--local_ip_for_outbound_sockets=127.22.37.2
--tserver_master_addrs=127.22.37.62:44233
--webserver_port=46767
--webserver_interface=127.22.37.2
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.2.keytab
--principal=kudu/127.22.37.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:37441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.22.37.20:58809/kms with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 18:38:41 dist-test-slave-5lgp krb5kdc[2061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980721, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.1@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM
I20260620 18:38:41.678507 3272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:41.630910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.1:55867 (local address 127.22.37.62:44233)
0620 18:38:41.633901 (+ 2991us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:41.633931 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:41.647187 (+ 13256us) server_negotiation.cc:407] Connection header received
0620 18:38:41.655321 (+ 8134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:41.655348 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:41.656007 (+ 659us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:41.656525 (+ 518us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:41.658576 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:41.660829 (+ 2253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:41.663254 (+ 2425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:41.664264 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:41.664424 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:41.669041 (+ 4617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:41.669053 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:41.669075 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:41.669126 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:41.673128 (+ 4002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:41.674103 (+ 975us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:41.674116 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:41.674123 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:41.674322 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:41.674861 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:41.674866 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:41.674869 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:41.675547 (+ 678us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:41.675739 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:41.676774 (+ 1035us) server_negotiation.cc:299] Negotiation successful
0620 18:38:41.677194 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2717,"thread_start_us":2535,"threads_started":1}
I20260620 18:38:41.683068 3273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:41.630927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:44233 (local address 127.22.37.1:55867)
0620 18:38:41.646939 (+ 16012us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:41.647008 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:41.650688 (+ 3680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:41.656688 (+ 6000us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:41.656701 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:41.657498 (+ 797us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:41.658309 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:41.658367 (+ 58us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:41.661046 (+ 2679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:41.661051 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:41.663086 (+ 2035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:41.663103 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:41.664854 (+ 1751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:41.664861 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:41.665133 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:41.665983 (+ 850us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:41.666013 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:41.668755 (+ 2742us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:41.673312 (+ 4557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:41.673324 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:41.673340 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:41.673928 (+ 588us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:41.674471 (+ 543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:41.674478 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:41.674483 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:41.674729 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:41.675830 (+ 1101us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:41.675837 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:41.676226 (+ 389us) client_negotiation.cc:769] Sending connection context
0620 18:38:41.678040 (+ 1814us) client_negotiation.cc:240] Negotiation successful
0620 18:38:41.681930 (+ 3890us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14776,"thread_start_us":14554,"threads_started":1}
I20260620 18:38:41.686655 3270 heartbeater.cc:347] Connected to a master server at 127.22.37.62:44233
I20260620 18:38:41.686988 3270 heartbeater.cc:464] Registering TS with master...
I20260620 18:38:41.687884 3270 heartbeater.cc:511] Master 127.22.37.62:44233 requested a full tablet report, sending...
W20260620 18:38:41.691118 3270 heartbeater.cc:644] Failed to heartbeat to 127.22.37.62:44233 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 18:38:41.691356 3270 heartbeater.cc:464] Registering TS with master...
I20260620 18:38:41.691653 3270 heartbeater.cc:511] Master 127.22.37.62:44233 requested a full tablet report, sending...
I20260620 18:38:41.692883 3270 heartbeater.cc:464] Registering TS with master...
I20260620 18:38:41.693248 3270 heartbeater.cc:511] Master 127.22.37.62:44233 requested a full tablet report, sending...
W20260620 18:38:41.694495 3270 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 18:38:42.041806 3274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:42.042218 3274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:42.042423 3274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:42.054150 3274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:42.054373 3274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:42.054600 3274 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.37.2
I20260620 18:38:42.070595 3274 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:37441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.22.37.20:58809/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.2.keytab
--principal=kudu/127.22.37.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.37.2:39873
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.37.2
--webserver_port=46767
--tserver_master_addrs=127.22.37.62:44233
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.37.2
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:38:42.072472 3274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:42.074234 3274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:42.090299 3290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:42.091395 3291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:42.093537 3293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:42.093797 3274 server_base.cc:1034] running on GCE node
I20260620 18:38:42.094576 3274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:42.095752 3274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:42.097019 3274 hybrid_clock.cc:648] HybridClock initialized: now 1781980722096951 us; error 63 us; skew 500 ppm
I20260620 18:38:42.097355 3274 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:38:42 dist-test-slave-5lgp krb5kdc[2061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980722, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:42.100658 3274 init.cc:377] Logged in from keytab as kudu/127.22.37.2@KRBTEST.COM (short username kudu)
I20260620 18:38:42.102557 3274 webserver.cc:460] Webserver started at http://127.22.37.2:46767/ using document root <none> and password file <none>
I20260620 18:38:42.103324 3274 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:42.103494 3274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 18:38:42 dist-test-slave-5lgp krb5kdc[2061](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781980722, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 18:38:42 dist-test-slave-5lgp krb5kdc[2061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980722, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.2@KRBTEST.COM for HTTP/127.22.37.20@KRBTEST.COM
I20260620 18:38:42.135058 3274 fs_manager.cc:709] Time spent opening directory manager: real 0.013s user 0.002s sys 0.005s
I20260620 18:38:42.141541 3301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:42.143229 3274 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.003s sys 0.000s
I20260620 18:38:42.143430 3274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ts-1/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ts-1/wal
uuid: "407863940edc4c2c86cc1c73bb877fbb"
format_stamp: "Formatted at 2026-06-20 18:38:39 on dist-test-slave-5lgp"
server_key: "c8bbe857e1f923a5e9ec31a92d688473"
server_key_iv: "0ad05ce85afd1fe49aeb042e0ff7250f"
server_key_version: "kuduclusterkey@0"
I20260620 18:38:42.144793 3274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:42.161726 3274 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:42.165894 3274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:42.166183 3274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:42.167248 3274 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:38:42.168875 3274 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:38:42.168992 3274 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:42.169123 3274 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:38:42.169212 3274 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:42.227258 3274 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.2:39873
I20260620 18:38:42.227317 3414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.2:39873 every 8 connection(s)
I20260620 18:38:42.229128 3274 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ts-1/data/info.pb
I20260620 18:38:42.233143 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 3274
I20260620 18:38:42.234092 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.37.20:58809/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.37.3:35693
--local_ip_for_outbound_sockets=127.22.37.3
--tserver_master_addrs=127.22.37.62:44233
--webserver_port=38561
--webserver_interface=127.22.37.3
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.3.keytab
--principal=kudu/127.22.37.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:37441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.22.37.20:58809/kms with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 18:38:42 dist-test-slave-5lgp krb5kdc[2061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980722, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.2@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM
I20260620 18:38:42.258203 3417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:42.232071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.2:55827 (local address 127.22.37.62:44233)
0620 18:38:42.232599 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:42.232606 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:42.233749 (+ 1143us) server_negotiation.cc:407] Connection header received
0620 18:38:42.237129 (+ 3380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:42.237138 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:42.237237 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:42.237325 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:42.239674 (+ 2349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:42.240883 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:42.243521 (+ 2638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:42.244305 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:42.244452 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:42.249882 (+ 5430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:42.249905 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:42.249913 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:42.249963 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:42.253879 (+ 3916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:42.254957 (+ 1078us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:42.254966 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:42.254972 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:42.255145 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:42.255755 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:42.255763 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:42.255767 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:42.256213 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:42.256365 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:42.257527 (+ 1162us) server_negotiation.cc:299] Negotiation successful
0620 18:38:42.257772 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":276,"threads_started":1}
I20260620 18:38:42.259729 3418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:42.232577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:44233 (local address 127.22.37.2:55827)
0620 18:38:42.233560 (+ 983us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:42.233619 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:42.236754 (+ 3135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:42.237948 (+ 1194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:42.237962 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:42.238622 (+ 660us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:42.239459 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:42.239488 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:42.241109 (+ 1621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:42.241117 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:42.243371 (+ 2254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:42.243389 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:42.244978 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:42.244987 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:42.245610 (+ 623us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:42.246465 (+ 855us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:42.246500 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:42.249591 (+ 3091us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:42.254072 (+ 4481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:42.254083 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:42.254103 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:42.254780 (+ 677us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:42.255309 (+ 529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:42.255317 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:42.255323 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:42.255584 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:42.256472 (+ 888us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:42.256483 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:42.256993 (+ 510us) client_negotiation.cc:769] Sending connection context
0620 18:38:42.257347 (+ 354us) client_negotiation.cc:240] Negotiation successful
0620 18:38:42.258076 (+ 729us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":383,"threads_started":1}
I20260620 18:38:42.261900 3415 heartbeater.cc:347] Connected to a master server at 127.22.37.62:44233
I20260620 18:38:42.262243 3415 heartbeater.cc:464] Registering TS with master...
I20260620 18:38:42.263180 3415 heartbeater.cc:511] Master 127.22.37.62:44233 requested a full tablet report, sending...
I20260620 18:38:42.265475 3072 ts_manager.cc:194] Registered new tserver with Master: 407863940edc4c2c86cc1c73bb877fbb (127.22.37.2:39873)
I20260620 18:38:42.267720 3072 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.37.2@KRBTEST.COM'} at 127.22.37.2:55827
Jun 20 18:38:42 dist-test-slave-5lgp krb5kdc[2061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980722, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.62@KRBTEST.COM for HTTP/127.22.37.20@KRBTEST.COM
W20260620 18:38:42.683158 3419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:42.683517 3419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:42.683705 3419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:42.694528 3419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
I20260620 18:38:42.694650 3270 heartbeater.cc:464] Registering TS with master...
W20260620 18:38:42.694698 3419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:42.694859 3419 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.37.3
I20260620 18:38:42.695086 3270 heartbeater.cc:511] Master 127.22.37.62:44233 requested a full tablet report, sending...
I20260620 18:38:42.696233 3072 ts_manager.cc:194] Registered new tserver with Master: c3c2ded58c2548c8a986e4af5cfee1d4 (127.22.37.1:41815)
I20260620 18:38:42.697583 3072 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.37.1@KRBTEST.COM'} at 127.22.37.1:55867
I20260620 18:38:42.709676 3419 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:37441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.22.37.20:58809/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.3.keytab
--principal=kudu/127.22.37.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.37.3:35693
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.37.3
--webserver_port=38561
--tserver_master_addrs=127.22.37.62:44233
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.37.3
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:38:42.711210 3419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:42.712842 3419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:42.728605 3424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:42.730119 3425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:42.745038 3427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:42.745522 3419 server_base.cc:1034] running on GCE node
I20260620 18:38:42.746826 3419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:42.748019 3419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:42.749300 3419 hybrid_clock.cc:648] HybridClock initialized: now 1781980722749220 us; error 71 us; skew 500 ppm
I20260620 18:38:42.749763 3419 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:38:42 dist-test-slave-5lgp krb5kdc[2061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980722, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:42.753654 3419 init.cc:377] Logged in from keytab as kudu/127.22.37.3@KRBTEST.COM (short username kudu)
I20260620 18:38:42.762171 3419 webserver.cc:460] Webserver started at http://127.22.37.3:38561/ using document root <none> and password file <none>
I20260620 18:38:42.763186 3419 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:42.763417 3419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 18:38:42 dist-test-slave-5lgp krb5kdc[2061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980722, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.3@KRBTEST.COM for HTTP/127.22.37.20@KRBTEST.COM
Jun 20 18:38:42 dist-test-slave-5lgp krb5kdc[2061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980722, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.62@KRBTEST.COM for HTTP/127.22.37.20@KRBTEST.COM
I20260620 18:38:42.802376 3419 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.003s sys 0.001s
I20260620 18:38:42.812367 3436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:42.814508 3419 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.006s sys 0.003s
I20260620 18:38:42.814728 3419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ts-2/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ts-2/wal
uuid: "8dba65949de942e3ac17753e65a8aa19"
format_stamp: "Formatted at 2026-06-20 18:38:39 on dist-test-slave-5lgp"
server_key: "9acf77498e46438c691ac9cfac5e97cd"
server_key_iv: "6b7503ede98bbf113eadbe5f0288f148"
server_key_version: "kuduclusterkey@0"
I20260620 18:38:42.816072 3419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:42.898917 3419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:42.910917 3419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:42.911311 3419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:42.937000 3419 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:38:42.945680 3419 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:38:42.945899 3419 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:42.946075 3419 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:38:42.946197 3419 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:43.000310 3419 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.3:35693
I20260620 18:38:43.000370 3548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.3:35693 every 8 connection(s)
I20260620 18:38:43.002416 3419 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ts-2/data/info.pb
I20260620 18:38:43.012315 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 3419
Jun 20 18:38:43 dist-test-slave-5lgp krb5kdc[2061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980722, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.3@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM
I20260620 18:38:43.051826 3552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:43.013827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41106 (local address 127.22.37.62:44233)
0620 18:38:43.033000 (+ 19173us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:43.033006 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:43.033020 (+ 14us) server_negotiation.cc:407] Connection header received
0620 18:38:43.033063 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:43.033067 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:43.033140 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:43.033240 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:43.035437 (+ 2197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:43.036634 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:43.039688 (+ 3054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:43.040303 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:43.040426 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:43.043980 (+ 3554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:43.043986 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:43.043991 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:43.044031 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:43.047881 (+ 3850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:43.048716 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:43.048721 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:43.048723 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:43.048959 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:43.049882 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:43.049897 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:43.049901 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:43.050290 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:43.050590 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:43.051166 (+ 576us) server_negotiation.cc:299] Negotiation successful
0620 18:38:43.051420 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19062,"thread_start_us":18257,"threads_started":1}
I20260620 18:38:43.056512 3554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:43.007560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.3:52951 (local address 127.22.37.62:44233)
0620 18:38:43.032211 (+ 24651us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:43.032217 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:43.032238 (+ 21us) server_negotiation.cc:407] Connection header received
0620 18:38:43.032299 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:43.032304 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:43.032401 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:43.032506 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:43.038426 (+ 5920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:43.039566 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:43.042857 (+ 3291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:43.043463 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:43.043551 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:43.048941 (+ 5390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:43.048948 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:43.048953 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:43.049005 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:43.052634 (+ 3629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:43.054204 (+ 1570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:43.054208 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:43.054210 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:43.054365 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:43.054839 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:43.054844 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:43.054847 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:43.055101 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:43.055247 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:43.055937 (+ 690us) server_negotiation.cc:299] Negotiation successful
0620 18:38:43.056143 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24519,"thread_start_us":25177,"threads_started":1}
I20260620 18:38:43.057585 3551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:43.006890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:44233 (local address 127.22.37.3:52951)
0620 18:38:43.028750 (+ 21860us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:43.028818 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:43.031465 (+ 2647us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:43.032947 (+ 1482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:43.032957 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:43.037289 (+ 4332us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:43.038192 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:43.038220 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:43.040701 (+ 2481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:43.040706 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:43.042678 (+ 1972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:43.042693 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:43.044800 (+ 2107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:43.044811 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:43.045106 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:43.045811 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:43.045840 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:43.048651 (+ 2811us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:43.052755 (+ 4104us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:43.052764 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:43.052783 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:43.053333 (+ 550us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:43.054521 (+ 1188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:43.054525 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:43.054527 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:43.054716 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:43.055230 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:43.055238 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:43.055563 (+ 325us) client_negotiation.cc:769] Sending connection context
0620 18:38:43.055843 (+ 280us) client_negotiation.cc:240] Negotiation successful
0620 18:38:43.056387 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21470,"thread_start_us":22405,"threads_started":1}
I20260620 18:38:43.059046 3549 heartbeater.cc:347] Connected to a master server at 127.22.37.62:44233
I20260620 18:38:43.059361 3549 heartbeater.cc:464] Registering TS with master...
I20260620 18:38:43.060197 3549 heartbeater.cc:511] Master 127.22.37.62:44233 requested a full tablet report, sending...
I20260620 18:38:43.061771 3072 ts_manager.cc:194] Registered new tserver with Master: 8dba65949de942e3ac17753e65a8aa19 (127.22.37.3:35693)
I20260620 18:38:43.062551 3072 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.37.3@KRBTEST.COM'} at 127.22.37.3:52951
I20260620 18:38:43.066649 22676 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:38:43.084420 3554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:43.069777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41122 (local address 127.22.37.62:44233)
0620 18:38:43.069922 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:43.069926 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:43.070376 (+ 450us) server_negotiation.cc:407] Connection header received
0620 18:38:43.070557 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:43.070562 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:43.070630 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:43.070734 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:43.072627 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:43.073307 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:43.075195 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:43.075622 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:43.075712 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:43.078914 (+ 3202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:43.078918 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:43.078921 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:43.078948 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:43.081407 (+ 2459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:43.082149 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:43.082153 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:43.082155 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:43.082270 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:43.082634 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:43.082637 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:43.082639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:43.083009 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:43.083250 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:43.083793 (+ 543us) server_negotiation.cc:299] Negotiation successful
0620 18:38:43.084109 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 18:38:43.089541 3072 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41122:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:38:43.096913 3072 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:38:43.147068 3574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:43.110963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.2:39873 (local address 127.0.0.1:55206)
0620 18:38:43.135172 (+ 24209us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:43.135236 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:43.135458 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:43.137060 (+ 1602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:43.137071 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:43.137117 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:38:43.138280 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:43.138316 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:43.141155 (+ 2839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:43.141163 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:43.143702 (+ 2539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:43.143718 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:43.146021 (+ 2303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:43.146028 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:43.146250 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:43.146309 (+ 59us) client_negotiation.cc:769] Sending connection context
0620 18:38:43.146474 (+ 165us) client_negotiation.cc:240] Negotiation successful
0620 18:38:43.146693 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23875,"thread_start_us":33332,"threads_started":1}
I20260620 18:38:43.148097 3573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:43.111349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55206 (local address 127.22.37.2:39873)
0620 18:38:43.136068 (+ 24719us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:43.136081 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:43.136118 (+ 37us) server_negotiation.cc:407] Connection header received
0620 18:38:43.136222 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:43.136231 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:43.136544 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:43.136758 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:38:43.138528 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:43.140958 (+ 2430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:43.143905 (+ 2947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:43.145456 (+ 1551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:43.145592 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:43.145820 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:43.147603 (+ 1783us) server_negotiation.cc:299] Negotiation successful
0620 18:38:43.147775 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24558,"thread_start_us":24456,"threads_started":1}
I20260620 18:38:43.152468 3350 tablet_service.cc:1467] Processing CreateTablet for tablet 623d3f0965b74ff5aec896bf1134f9a3 (DEFAULT_TABLE table=test-table [id=93384f0e7b3c464594eb3b766aafc755]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:38:43.153357 3571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:43.111034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.3:35693 (local address 127.0.0.1:48516)
0620 18:38:43.144002 (+ 32968us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:43.144028 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:43.144180 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:43.146231 (+ 2051us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:43.146235 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:43.146262 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:38:43.146668 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:43.146680 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:43.148695 (+ 2015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:43.148699 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:43.150736 (+ 2037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:43.150750 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:43.152529 (+ 1779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:43.152536 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:43.152693 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:43.152727 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 18:38:43.152817 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 18:38:43.152947 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32794,"mutex_wait_us":76,"thread_start_us":34764,"threads_started":1}
I20260620 18:38:43.153451 3575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:43.111453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48516 (local address 127.22.37.3:35693)
0620 18:38:43.139572 (+ 28119us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:43.139583 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:43.144580 (+ 4997us) server_negotiation.cc:407] Connection header received
0620 18:38:43.144714 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:43.144725 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:43.145059 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:43.145305 (+ 246us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:38:43.146906 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:43.148515 (+ 1609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:43.150925 (+ 2410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:43.152316 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:43.152469 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:43.152704 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:43.152892 (+ 188us) server_negotiation.cc:299] Negotiation successful
0620 18:38:43.153051 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27922,"thread_start_us":27786,"threads_started":1}
I20260620 18:38:43.154318 3572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:43.111345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.1:41815 (local address 127.0.0.1:56652)
0620 18:38:43.146136 (+ 34791us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:43.146155 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:43.146397 (+ 242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:43.147360 (+ 963us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:43.147365 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:43.147396 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:38:43.147735 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:43.147747 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:43.149685 (+ 1938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:43.149690 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:43.151631 (+ 1941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:43.151642 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:43.153454 (+ 1812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:43.153462 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:43.153699 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:43.153735 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 18:38:43.153851 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 18:38:43.153978 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34633,"mutex_wait_us":4,"thread_start_us":26048,"threads_started":1}
I20260620 18:38:43.154336 3350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 623d3f0965b74ff5aec896bf1134f9a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:43.154382 3576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:43.112029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56652 (local address 127.22.37.1:41815)
0620 18:38:43.134263 (+ 22234us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:43.134276 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:43.146268 (+ 11992us) server_negotiation.cc:407] Connection header received
0620 18:38:43.146607 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:43.146615 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:43.146855 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:43.147030 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:38:43.147911 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:43.149563 (+ 1652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:43.151780 (+ 2217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:43.152945 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:43.153066 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:43.153268 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:43.153913 (+ 645us) server_negotiation.cc:299] Negotiation successful
0620 18:38:43.154068 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21992,"thread_start_us":22242,"threads_started":1}
I20260620 18:38:43.157411 3205 tablet_service.cc:1467] Processing CreateTablet for tablet 623d3f0965b74ff5aec896bf1134f9a3 (DEFAULT_TABLE table=test-table [id=93384f0e7b3c464594eb3b766aafc755]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:38:43.157289 3484 tablet_service.cc:1467] Processing CreateTablet for tablet 623d3f0965b74ff5aec896bf1134f9a3 (DEFAULT_TABLE table=test-table [id=93384f0e7b3c464594eb3b766aafc755]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:38:43.158553 3205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 623d3f0965b74ff5aec896bf1134f9a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:43.158676 3484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 623d3f0965b74ff5aec896bf1134f9a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:43.169871 3577 tablet_bootstrap.cc:492] T 623d3f0965b74ff5aec896bf1134f9a3 P c3c2ded58c2548c8a986e4af5cfee1d4: Bootstrap starting.
I20260620 18:38:43.173019 3577 tablet_bootstrap.cc:654] T 623d3f0965b74ff5aec896bf1134f9a3 P c3c2ded58c2548c8a986e4af5cfee1d4: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:43.173738 3579 tablet_bootstrap.cc:492] T 623d3f0965b74ff5aec896bf1134f9a3 P 8dba65949de942e3ac17753e65a8aa19: Bootstrap starting.
I20260620 18:38:43.174705 3578 tablet_bootstrap.cc:492] T 623d3f0965b74ff5aec896bf1134f9a3 P 407863940edc4c2c86cc1c73bb877fbb: Bootstrap starting.
I20260620 18:38:43.175511 3577 log.cc:824] T 623d3f0965b74ff5aec896bf1134f9a3 P c3c2ded58c2548c8a986e4af5cfee1d4: Log is configured to *not* fsync() on all Append() calls
I20260620 18:38:43.177327 3578 tablet_bootstrap.cc:654] T 623d3f0965b74ff5aec896bf1134f9a3 P 407863940edc4c2c86cc1c73bb877fbb: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:43.178444 3579 tablet_bootstrap.cc:654] T 623d3f0965b74ff5aec896bf1134f9a3 P 8dba65949de942e3ac17753e65a8aa19: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:43.178491 3578 log.cc:824] T 623d3f0965b74ff5aec896bf1134f9a3 P 407863940edc4c2c86cc1c73bb877fbb: Log is configured to *not* fsync() on all Append() calls
I20260620 18:38:43.179553 3579 log.cc:824] T 623d3f0965b74ff5aec896bf1134f9a3 P 8dba65949de942e3ac17753e65a8aa19: Log is configured to *not* fsync() on all Append() calls
I20260620 18:38:43.180063 3577 tablet_bootstrap.cc:492] T 623d3f0965b74ff5aec896bf1134f9a3 P c3c2ded58c2548c8a986e4af5cfee1d4: No bootstrap required, opened a new log
I20260620 18:38:43.180375 3577 ts_tablet_manager.cc:1397] T 623d3f0965b74ff5aec896bf1134f9a3 P c3c2ded58c2548c8a986e4af5cfee1d4: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260620 18:38:43.181097 3578 tablet_bootstrap.cc:492] T 623d3f0965b74ff5aec896bf1134f9a3 P 407863940edc4c2c86cc1c73bb877fbb: No bootstrap required, opened a new log
I20260620 18:38:43.181339 3578 ts_tablet_manager.cc:1397] T 623d3f0965b74ff5aec896bf1134f9a3 P 407863940edc4c2c86cc1c73bb877fbb: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260620 18:38:43.182446 3579 tablet_bootstrap.cc:492] T 623d3f0965b74ff5aec896bf1134f9a3 P 8dba65949de942e3ac17753e65a8aa19: No bootstrap required, opened a new log
I20260620 18:38:43.182686 3579 ts_tablet_manager.cc:1397] T 623d3f0965b74ff5aec896bf1134f9a3 P 8dba65949de942e3ac17753e65a8aa19: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260620 18:38:43.188215 3578 raft_consensus.cc:348] T 623d3f0965b74ff5aec896bf1134f9a3 P 407863940edc4c2c86cc1c73bb877fbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dba65949de942e3ac17753e65a8aa19" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35693 } } peers { permanent_uuid: "407863940edc4c2c86cc1c73bb877fbb" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 39873 } } peers { permanent_uuid: "c3c2ded58c2548c8a986e4af5cfee1d4" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 41815 } }
I20260620 18:38:43.188581 3578 raft_consensus.cc:374] T 623d3f0965b74ff5aec896bf1134f9a3 P 407863940edc4c2c86cc1c73bb877fbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:43.188727 3578 raft_consensus.cc:729] T 623d3f0965b74ff5aec896bf1134f9a3 P 407863940edc4c2c86cc1c73bb877fbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 407863940edc4c2c86cc1c73bb877fbb, State: Initialized, Role: FOLLOWER
I20260620 18:38:43.189340 3578 consensus_queue.cc:260] T 623d3f0965b74ff5aec896bf1134f9a3 P 407863940edc4c2c86cc1c73bb877fbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dba65949de942e3ac17753e65a8aa19" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35693 } } peers { permanent_uuid: "407863940edc4c2c86cc1c73bb877fbb" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 39873 } } peers { permanent_uuid: "c3c2ded58c2548c8a986e4af5cfee1d4" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 41815 } }
I20260620 18:38:43.190629 3415 heartbeater.cc:503] Master 127.22.37.62:44233 was elected leader, sending a full tablet report...
I20260620 18:38:43.190819 3577 raft_consensus.cc:348] T 623d3f0965b74ff5aec896bf1134f9a3 P c3c2ded58c2548c8a986e4af5cfee1d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dba65949de942e3ac17753e65a8aa19" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35693 } } peers { permanent_uuid: "407863940edc4c2c86cc1c73bb877fbb" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 39873 } } peers { permanent_uuid: "c3c2ded58c2548c8a986e4af5cfee1d4" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 41815 } }
I20260620 18:38:43.191201 3577 raft_consensus.cc:374] T 623d3f0965b74ff5aec896bf1134f9a3 P c3c2ded58c2548c8a986e4af5cfee1d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:43.191287 3577 raft_consensus.cc:729] T 623d3f0965b74ff5aec896bf1134f9a3 P c3c2ded58c2548c8a986e4af5cfee1d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3c2ded58c2548c8a986e4af5cfee1d4, State: Initialized, Role: FOLLOWER
I20260620 18:38:43.191334 3578 ts_tablet_manager.cc:1428] T 623d3f0965b74ff5aec896bf1134f9a3 P 407863940edc4c2c86cc1c73bb877fbb: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20260620 18:38:43.191814 3577 consensus_queue.cc:260] T 623d3f0965b74ff5aec896bf1134f9a3 P c3c2ded58c2548c8a986e4af5cfee1d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dba65949de942e3ac17753e65a8aa19" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35693 } } peers { permanent_uuid: "407863940edc4c2c86cc1c73bb877fbb" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 39873 } } peers { permanent_uuid: "c3c2ded58c2548c8a986e4af5cfee1d4" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 41815 } }
I20260620 18:38:43.192865 3270 heartbeater.cc:503] Master 127.22.37.62:44233 was elected leader, sending a full tablet report...
I20260620 18:38:43.193430 3577 ts_tablet_manager.cc:1428] T 623d3f0965b74ff5aec896bf1134f9a3 P c3c2ded58c2548c8a986e4af5cfee1d4: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260620 18:38:43.194036 3579 raft_consensus.cc:348] T 623d3f0965b74ff5aec896bf1134f9a3 P 8dba65949de942e3ac17753e65a8aa19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dba65949de942e3ac17753e65a8aa19" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35693 } } peers { permanent_uuid: "407863940edc4c2c86cc1c73bb877fbb" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 39873 } } peers { permanent_uuid: "c3c2ded58c2548c8a986e4af5cfee1d4" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 41815 } }
I20260620 18:38:43.194418 3579 raft_consensus.cc:374] T 623d3f0965b74ff5aec896bf1134f9a3 P 8dba65949de942e3ac17753e65a8aa19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:43.194515 3579 raft_consensus.cc:729] T 623d3f0965b74ff5aec896bf1134f9a3 P 8dba65949de942e3ac17753e65a8aa19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dba65949de942e3ac17753e65a8aa19, State: Initialized, Role: FOLLOWER
I20260620 18:38:43.195103 3579 consensus_queue.cc:260] T 623d3f0965b74ff5aec896bf1134f9a3 P 8dba65949de942e3ac17753e65a8aa19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dba65949de942e3ac17753e65a8aa19" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35693 } } peers { permanent_uuid: "407863940edc4c2c86cc1c73bb877fbb" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 39873 } } peers { permanent_uuid: "c3c2ded58c2548c8a986e4af5cfee1d4" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 41815 } }
I20260620 18:38:43.196164 3549 heartbeater.cc:503] Master 127.22.37.62:44233 was elected leader, sending a full tablet report...
I20260620 18:38:43.196838 3579 ts_tablet_manager.cc:1428] T 623d3f0965b74ff5aec896bf1134f9a3 P 8dba65949de942e3ac17753e65a8aa19: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
W20260620 18:38:43.231370 3416 tablet.cc:2369] T 623d3f0965b74ff5aec896bf1134f9a3 P 407863940edc4c2c86cc1c73bb877fbb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:38:43.256292 3550 tablet.cc:2369] T 623d3f0965b74ff5aec896bf1134f9a3 P 8dba65949de942e3ac17753e65a8aa19: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:38:43.380537 3271 tablet.cc:2369] T 623d3f0965b74ff5aec896bf1134f9a3 P c3c2ded58c2548c8a986e4af5cfee1d4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:38:43.387607 3584 raft_consensus.cc:482] T 623d3f0965b74ff5aec896bf1134f9a3 P c3c2ded58c2548c8a986e4af5cfee1d4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:38:43.387926 3584 raft_consensus.cc:504] T 623d3f0965b74ff5aec896bf1134f9a3 P c3c2ded58c2548c8a986e4af5cfee1d4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dba65949de942e3ac17753e65a8aa19" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35693 } } peers { permanent_uuid: "407863940edc4c2c86cc1c73bb877fbb" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 39873 } } peers { permanent_uuid: "c3c2ded58c2548c8a986e4af5cfee1d4" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 41815 } }
I20260620 18:38:43.389740 3584 leader_election.cc:288] T 623d3f0965b74ff5aec896bf1134f9a3 P c3c2ded58c2548c8a986e4af5cfee1d4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8dba65949de942e3ac17753e65a8aa19 (127.22.37.3:35693), 407863940edc4c2c86cc1c73bb877fbb (127.22.37.2:39873)
I20260620 18:38:43.396692 3573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:43.390890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.1:54737 (local address 127.22.37.2:39873)
0620 18:38:43.391092 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:43.391099 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:43.391357 (+ 258us) server_negotiation.cc:407] Connection header received
0620 18:38:43.391452 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:43.391456 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:43.391663 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:43.391764 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:38:43.392353 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:43.393045 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:43.394929 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:43.395747 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:43.395853 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:43.395963 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:43.396290 (+ 327us) server_negotiation.cc:299] Negotiation successful
0620 18:38:43.396429 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 18:38:43.396728 3575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:43.390738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.1:51263 (local address 127.22.37.3:35693)
0620 18:38:43.390940 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:43.390949 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:43.391042 (+ 93us) server_negotiation.cc:407] Connection header received
0620 18:38:43.391326 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:43.391334 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:43.391420 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:43.391506 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:38:43.392285 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:43.393023 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:43.395018 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:43.395793 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:43.395955 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:43.396047 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:43.396374 (+ 327us) server_negotiation.cc:299] Negotiation successful
0620 18:38:43.396504 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260620 18:38:43.396790 3587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:43.390267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.2:39873 (local address 127.22.37.1:54737)
0620 18:38:43.391186 (+ 919us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:43.391200 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:43.391351 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:43.391799 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:43.391802 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:43.391846 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:38:43.392179 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:43.392186 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:43.393208 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:43.393212 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:43.394777 (+ 1565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:43.394788 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:43.395968 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:43.395974 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:43.396137 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:43.396160 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 18:38:43.396233 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 18:38:43.396374 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":790,"mutex_wait_us":1,"thread_start_us":798,"threads_started":1}
I20260620 18:38:43.396907 3586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:43.390187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.3:35693 (local address 127.22.37.1:51263)
0620 18:38:43.390907 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:43.390928 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:43.391131 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:43.391637 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:43.391643 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:43.391667 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:38:43.392132 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:43.392142 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:43.393149 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:43.393154 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:43.394844 (+ 1690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:43.394877 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:43.396056 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:43.396061 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:43.396184 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:43.396206 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 18:38:43.396354 (+ 148us) client_negotiation.cc:240] Negotiation successful
0620 18:38:43.396467 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"mutex_wait_us":23,"spinlock_wait_cycles":20864,"thread_start_us":374,"threads_started":1}
I20260620 18:38:43.398020 3504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "623d3f0965b74ff5aec896bf1134f9a3" candidate_uuid: "c3c2ded58c2548c8a986e4af5cfee1d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dba65949de942e3ac17753e65a8aa19" is_pre_election: true
I20260620 18:38:43.398144 3370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "623d3f0965b74ff5aec896bf1134f9a3" candidate_uuid: "c3c2ded58c2548c8a986e4af5cfee1d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "407863940edc4c2c86cc1c73bb877fbb" is_pre_election: true
I20260620 18:38:43.398514 3504 raft_consensus.cc:2454] T 623d3f0965b74ff5aec896bf1134f9a3 P 8dba65949de942e3ac17753e65a8aa19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3c2ded58c2548c8a986e4af5cfee1d4 in term 0.
I20260620 18:38:43.398697 3370 raft_consensus.cc:2454] T 623d3f0965b74ff5aec896bf1134f9a3 P 407863940edc4c2c86cc1c73bb877fbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3c2ded58c2548c8a986e4af5cfee1d4 in term 0.
I20260620 18:38:43.399190 3159 leader_election.cc:302] T 623d3f0965b74ff5aec896bf1134f9a3 P c3c2ded58c2548c8a986e4af5cfee1d4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8dba65949de942e3ac17753e65a8aa19, c3c2ded58c2548c8a986e4af5cfee1d4; no voters:
I20260620 18:38:43.399569 3584 raft_consensus.cc:2781] T 623d3f0965b74ff5aec896bf1134f9a3 P c3c2ded58c2548c8a986e4af5cfee1d4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:38:43.399721 3584 raft_consensus.cc:482] T 623d3f0965b74ff5aec896bf1134f9a3 P c3c2ded58c2548c8a986e4af5cfee1d4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:38:43.399833 3584 raft_consensus.cc:3037] T 623d3f0965b74ff5aec896bf1134f9a3 P c3c2ded58c2548c8a986e4af5cfee1d4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:43.402081 3584 raft_consensus.cc:504] T 623d3f0965b74ff5aec896bf1134f9a3 P c3c2ded58c2548c8a986e4af5cfee1d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dba65949de942e3ac17753e65a8aa19" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35693 } } peers { permanent_uuid: "407863940edc4c2c86cc1c73bb877fbb" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 39873 } } peers { permanent_uuid: "c3c2ded58c2548c8a986e4af5cfee1d4" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 41815 } }
I20260620 18:38:43.402664 3584 leader_election.cc:288] T 623d3f0965b74ff5aec896bf1134f9a3 P c3c2ded58c2548c8a986e4af5cfee1d4 [CANDIDATE]: Term 1 election: Requested vote from peers 8dba65949de942e3ac17753e65a8aa19 (127.22.37.3:35693), 407863940edc4c2c86cc1c73bb877fbb (127.22.37.2:39873)
I20260620 18:38:43.403128 3370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "623d3f0965b74ff5aec896bf1134f9a3" candidate_uuid: "c3c2ded58c2548c8a986e4af5cfee1d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "407863940edc4c2c86cc1c73bb877fbb"
I20260620 18:38:43.403162 3504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "623d3f0965b74ff5aec896bf1134f9a3" candidate_uuid: "c3c2ded58c2548c8a986e4af5cfee1d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dba65949de942e3ac17753e65a8aa19"
I20260620 18:38:43.403355 3370 raft_consensus.cc:3037] T 623d3f0965b74ff5aec896bf1134f9a3 P 407863940edc4c2c86cc1c73bb877fbb [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:43.403357 3504 raft_consensus.cc:3037] T 623d3f0965b74ff5aec896bf1134f9a3 P 8dba65949de942e3ac17753e65a8aa19 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:43.406457 3370 raft_consensus.cc:2454] T 623d3f0965b74ff5aec896bf1134f9a3 P 407863940edc4c2c86cc1c73bb877fbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3c2ded58c2548c8a986e4af5cfee1d4 in term 1.
I20260620 18:38:43.406457 3504 raft_consensus.cc:2454] T 623d3f0965b74ff5aec896bf1134f9a3 P 8dba65949de942e3ac17753e65a8aa19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3c2ded58c2548c8a986e4af5cfee1d4 in term 1.
I20260620 18:38:43.406852 3159 leader_election.cc:302] T 623d3f0965b74ff5aec896bf1134f9a3 P c3c2ded58c2548c8a986e4af5cfee1d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8dba65949de942e3ac17753e65a8aa19, c3c2ded58c2548c8a986e4af5cfee1d4; no voters:
I20260620 18:38:43.407156 3584 raft_consensus.cc:2781] T 623d3f0965b74ff5aec896bf1134f9a3 P c3c2ded58c2548c8a986e4af5cfee1d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:38:43.407652 3584 raft_consensus.cc:686] T 623d3f0965b74ff5aec896bf1134f9a3 P c3c2ded58c2548c8a986e4af5cfee1d4 [term 1 LEADER]: Becoming Leader. State: Replica: c3c2ded58c2548c8a986e4af5cfee1d4, State: Running, Role: LEADER
I20260620 18:38:43.408140 3584 consensus_queue.cc:237] T 623d3f0965b74ff5aec896bf1134f9a3 P c3c2ded58c2548c8a986e4af5cfee1d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dba65949de942e3ac17753e65a8aa19" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35693 } } peers { permanent_uuid: "407863940edc4c2c86cc1c73bb877fbb" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 39873 } } peers { permanent_uuid: "c3c2ded58c2548c8a986e4af5cfee1d4" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 41815 } }
I20260620 18:38:43.412474 3072 catalog_manager.cc:5495] T 623d3f0965b74ff5aec896bf1134f9a3 P c3c2ded58c2548c8a986e4af5cfee1d4 reported cstate change: term changed from 0 to 1, leader changed from <none> to c3c2ded58c2548c8a986e4af5cfee1d4 (127.22.37.1). New cstate: current_term: 1 leader_uuid: "c3c2ded58c2548c8a986e4af5cfee1d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dba65949de942e3ac17753e65a8aa19" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "407863940edc4c2c86cc1c73bb877fbb" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 39873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3c2ded58c2548c8a986e4af5cfee1d4" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 41815 } health_report { overall_health: HEALTHY } } }
I20260620 18:38:43.487347 3576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:43.481827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56654 (local address 127.22.37.1:41815)
0620 18:38:43.482029 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:43.482036 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:43.482100 (+ 64us) server_negotiation.cc:407] Connection header received
0620 18:38:43.482391 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:43.482395 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:43.482493 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:43.482638 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:38:43.483048 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:43.483789 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:43.485191 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:43.485635 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:43.485758 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:43.486277 (+ 519us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:38:43.486788 (+ 511us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:38:43.486949 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:43.487004 (+ 55us) server_negotiation.cc:299] Negotiation successful
0620 18:38:43.487082 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 18:38:43.495244 3370 raft_consensus.cc:1261] T 623d3f0965b74ff5aec896bf1134f9a3 P 407863940edc4c2c86cc1c73bb877fbb [term 1 FOLLOWER]: Refusing update from remote peer c3c2ded58c2548c8a986e4af5cfee1d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:38:43.495242 3504 raft_consensus.cc:1261] T 623d3f0965b74ff5aec896bf1134f9a3 P 8dba65949de942e3ac17753e65a8aa19 [term 1 FOLLOWER]: Refusing update from remote peer c3c2ded58c2548c8a986e4af5cfee1d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:38:43.496100 3588 consensus_queue.cc:1035] T 623d3f0965b74ff5aec896bf1134f9a3 P c3c2ded58c2548c8a986e4af5cfee1d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dba65949de942e3ac17753e65a8aa19" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:38:43.496402 3584 consensus_queue.cc:1035] T 623d3f0965b74ff5aec896bf1134f9a3 P c3c2ded58c2548c8a986e4af5cfee1d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "407863940edc4c2c86cc1c73bb877fbb" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 39873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:38:43.507422 3591 mvcc.cc:205] Tried to move back new op lower bound from 7298993043427262464 to 7298993043083550720. Current Snapshot: MvccSnapshot[applied={T|T < 7298993043427262464}]
I20260620 18:38:43.521311 3071 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41122:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:38:43.521705 3071 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41122:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:38:43.525126 3071 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6893290e7c064c4eb0cf819f9ccbe23b: Sending DeleteTablet for 3 replicas of tablet 623d3f0965b74ff5aec896bf1134f9a3
I20260620 18:38:43.526335 3350 tablet_service.cc:1514] Processing DeleteTablet for tablet 623d3f0965b74ff5aec896bf1134f9a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:38:43 UTC) from {username='kudu', principal='kudu/127.22.37.62@KRBTEST.COM'} at 127.0.0.1:55206
I20260620 18:38:43.526356 3484 tablet_service.cc:1514] Processing DeleteTablet for tablet 623d3f0965b74ff5aec896bf1134f9a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:38:43 UTC) from {username='kudu', principal='kudu/127.22.37.62@KRBTEST.COM'} at 127.0.0.1:48516
I20260620 18:38:43.527035 3205 tablet_service.cc:1514] Processing DeleteTablet for tablet 623d3f0965b74ff5aec896bf1134f9a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:38:43 UTC) from {username='kudu', principal='kudu/127.22.37.62@KRBTEST.COM'} at 127.0.0.1:56652
I20260620 18:38:43.533906 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 3109
I20260620 18:38:43.536302 3603 tablet_replica.cc:330] T 623d3f0965b74ff5aec896bf1134f9a3 P c3c2ded58c2548c8a986e4af5cfee1d4: stopping tablet replica
I20260620 18:38:43.536972 3603 raft_consensus.cc:2229] T 623d3f0965b74ff5aec896bf1134f9a3 P c3c2ded58c2548c8a986e4af5cfee1d4 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:38:43.537998 3603 raft_consensus.cc:2258] T 623d3f0965b74ff5aec896bf1134f9a3 P c3c2ded58c2548c8a986e4af5cfee1d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:38:43.540885 3603 ts_tablet_manager.cc:1905] T 623d3f0965b74ff5aec896bf1134f9a3 P c3c2ded58c2548c8a986e4af5cfee1d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:38:43.551010 3603 ts_tablet_manager.cc:1918] T 623d3f0965b74ff5aec896bf1134f9a3 P c3c2ded58c2548c8a986e4af5cfee1d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:38:43.551333 3603 log.cc:1196] T 623d3f0965b74ff5aec896bf1134f9a3 P c3c2ded58c2548c8a986e4af5cfee1d4: Deleting WAL directory at /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ts-0/wal/wals/623d3f0965b74ff5aec896bf1134f9a3
I20260620 18:38:43.552206 3601 tablet_replica.cc:330] T 623d3f0965b74ff5aec896bf1134f9a3 P 8dba65949de942e3ac17753e65a8aa19: stopping tablet replica
I20260620 18:38:43.552667 3601 raft_consensus.cc:2229] T 623d3f0965b74ff5aec896bf1134f9a3 P 8dba65949de942e3ac17753e65a8aa19 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:38:43.553135 3602 tablet_replica.cc:330] T 623d3f0965b74ff5aec896bf1134f9a3 P 407863940edc4c2c86cc1c73bb877fbb: stopping tablet replica
I20260620 18:38:43.553308 3601 raft_consensus.cc:2258] T 623d3f0965b74ff5aec896bf1134f9a3 P 8dba65949de942e3ac17753e65a8aa19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:38:43.553612 3602 raft_consensus.cc:2229] T 623d3f0965b74ff5aec896bf1134f9a3 P 407863940edc4c2c86cc1c73bb877fbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:38:43.553771 3603 ts_tablet_manager.cc:1939] T 623d3f0965b74ff5aec896bf1134f9a3 P c3c2ded58c2548c8a986e4af5cfee1d4: Deleting consensus metadata
I20260620 18:38:43.554159 3602 raft_consensus.cc:2258] T 623d3f0965b74ff5aec896bf1134f9a3 P 407863940edc4c2c86cc1c73bb877fbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:38:43.554518 3576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:43.534980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56658 (local address 127.22.37.1:41815)
0620 18:38:43.535165 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:43.535172 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:43.535953 (+ 781us) server_negotiation.cc:407] Connection header received
0620 18:38:43.536174 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:43.536179 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:43.536273 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:43.536394 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:43.538126 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:43.539351 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:43.541217 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:43.541917 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:43.542047 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:43.545238 (+ 3191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:43.545251 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:43.545268 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:43.545324 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:43.549727 (+ 4403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:43.550669 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:43.550678 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:43.550682 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:43.550846 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:43.551331 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:43.551338 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:43.551342 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:43.552168 (+ 826us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:43.552464 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:43.553821 (+ 1357us) server_negotiation.cc:299] Negotiation successful
0620 18:38:43.554061 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 18:38:43.555445 3265 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:38:43.555811 3601 ts_tablet_manager.cc:1905] T 623d3f0965b74ff5aec896bf1134f9a3 P 8dba65949de942e3ac17753e65a8aa19: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:38:43.555915 3059 catalog_manager.cc:4841] TS c3c2ded58c2548c8a986e4af5cfee1d4 (127.22.37.1:41815): tablet 623d3f0965b74ff5aec896bf1134f9a3 (table test-table [id=93384f0e7b3c464594eb3b766aafc755]) successfully deleted
I20260620 18:38:43.556406 3602 ts_tablet_manager.cc:1905] T 623d3f0965b74ff5aec896bf1134f9a3 P 407863940edc4c2c86cc1c73bb877fbb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:38:43.563066 3602 ts_tablet_manager.cc:1918] T 623d3f0965b74ff5aec896bf1134f9a3 P 407863940edc4c2c86cc1c73bb877fbb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:38:43.563256 3602 log.cc:1196] T 623d3f0965b74ff5aec896bf1134f9a3 P 407863940edc4c2c86cc1c73bb877fbb: Deleting WAL directory at /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ts-1/wal/wals/623d3f0965b74ff5aec896bf1134f9a3
I20260620 18:38:43.563314 3601 ts_tablet_manager.cc:1918] T 623d3f0965b74ff5aec896bf1134f9a3 P 8dba65949de942e3ac17753e65a8aa19: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:38:43.563490 3601 log.cc:1196] T 623d3f0965b74ff5aec896bf1134f9a3 P 8dba65949de942e3ac17753e65a8aa19: Deleting WAL directory at /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980527431848-22676-0/minicluster-data/ts-2/wal/wals/623d3f0965b74ff5aec896bf1134f9a3
I20260620 18:38:43.563838 3602 ts_tablet_manager.cc:1939] T 623d3f0965b74ff5aec896bf1134f9a3 P 407863940edc4c2c86cc1c73bb877fbb: Deleting consensus metadata
I20260620 18:38:43.564025 3601 ts_tablet_manager.cc:1939] T 623d3f0965b74ff5aec896bf1134f9a3 P 8dba65949de942e3ac17753e65a8aa19: Deleting consensus metadata
I20260620 18:38:43.565657 3056 catalog_manager.cc:4841] TS 407863940edc4c2c86cc1c73bb877fbb (127.22.37.2:39873): tablet 623d3f0965b74ff5aec896bf1134f9a3 (table test-table [id=93384f0e7b3c464594eb3b766aafc755]) successfully deleted
I20260620 18:38:43.565690 3057 catalog_manager.cc:4841] TS 8dba65949de942e3ac17753e65a8aa19 (127.22.37.3:35693): tablet 623d3f0965b74ff5aec896bf1134f9a3 (table test-table [id=93384f0e7b3c464594eb3b766aafc755]) successfully deleted
I20260620 18:38:43.655823 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 3109
I20260620 18:38:43.673213 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 3274
I20260620 18:38:43.686517 3573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:43.674114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55208 (local address 127.22.37.2:39873)
0620 18:38:43.674288 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:43.674292 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:43.674332 (+ 40us) server_negotiation.cc:407] Connection header received
0620 18:38:43.674465 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:43.674470 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:43.674533 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:43.674615 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:43.675902 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:43.676621 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:43.677847 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:43.678249 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:43.678382 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:43.680841 (+ 2459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:43.680856 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:43.680872 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:43.680911 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:43.683815 (+ 2904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:43.684496 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:43.684503 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:43.684505 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:43.684626 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:43.684981 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:43.684984 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:43.684986 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:43.685420 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:43.685659 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:43.685901 (+ 242us) server_negotiation.cc:299] Negotiation successful
0620 18:38:43.686074 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 18:38:43.687204 3410 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:38:43.776296 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 3274
I20260620 18:38:43.795204 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 3419
I20260620 18:38:43.809762 3575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:43.796085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48524 (local address 127.22.37.3:35693)
0620 18:38:43.796260 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:43.796267 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:43.796340 (+ 73us) server_negotiation.cc:407] Connection header received
0620 18:38:43.796582 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:43.796586 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:43.796651 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:43.796754 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:43.798130 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:43.798861 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:43.800204 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:43.800597 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:43.800733 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:43.803486 (+ 2753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:43.803503 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:43.803522 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:43.803560 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:43.806711 (+ 3151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:43.807533 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:43.807541 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:43.807544 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:43.807698 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:43.808111 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:43.808116 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:43.808120 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:43.808488 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:43.808831 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:43.809154 (+ 323us) server_negotiation.cc:299] Negotiation successful
0620 18:38:43.809409 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 18:38:43.810420 3544 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:38:43.899227 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 3419
I20260620 18:38:43.916462 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 3039
I20260620 18:38:43.917757 3102 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:38:44.029820 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 3039
I20260620 18:38:44.051656 22676 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 18:38:44.386641 22676 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 18:38:44.386876 22676 mini_ranger.cc:66] Stopping Ranger...
I20260620 18:38:44.731997 22676 mini_ranger.cc:68] Stopped Ranger
2026-06-20 18:38:44.732 UTC [2091] LOG: received smart shutdown request
2026-06-20 18:38:44.735 UTC [2091] LOG: background worker "logical replication launcher" (PID 2100) exited with exit code 1
2026-06-20 18:38:44.735 UTC [2095] LOG: shutting down
2026-06-20 18:38:44.896 UTC [2091] LOG: database system is shut down
2026-06-20T18:38:44Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (51568 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980527431848-22676-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:38:45 dist-test-slave-5lgp krb5kdc[3616](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:38:45 dist-test-slave-5lgp krb5kdc[3616](info): set up 2 sockets
Jun 20 18:38:45 dist-test-slave-5lgp krb5kdc[3616](info): commencing operation
krb5kdc: starting...
W20260620 18:38:46.141739 22676 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.009s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980527431848-22676-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980527431848-22676-0/krb5kdc/test-user.keytab.
Jun 20 18:38:46 dist-test-slave-5lgp krb5kdc[3616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980726, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:38:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:38:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.37.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.37.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.37.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab.
Entry for principal kudu/127.22.37.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab.
I20260620 18:38:46.365120 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980527431848-22676-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980527431848-22676-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980527431848-22676-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.62:40755
--webserver_interface=127.22.37.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab
--principal=kudu/127.22.37.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:34989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.37.62:40755,127.22.37.61:42843,127.22.37.60:35949
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:38:46.692687 3629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:46.693034 3629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:46.693185 3629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:46.700474 3629 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:38:46.700641 3629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:46.700744 3629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:46.700845 3629 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:38:46.700944 3629 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:38:46.711360 3629 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:34989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980527431848-22676-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980527431848-22676-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.37.62:40755,127.22.37.61:42843,127.22.37.60:35949
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab
--principal=kudu/127.22.37.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.62:40755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.37.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980527431848-22676-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:38:46.712795 3629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:46.714159 3629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:46.725924 3634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:46.726046 3635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:46.726106 3637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:46.727098 3629 server_base.cc:1034] running on GCE node
I20260620 18:38:46.727845 3629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:46.729171 3629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:46.730379 3629 hybrid_clock.cc:648] HybridClock initialized: now 1781980726730320 us; error 54 us; skew 500 ppm
I20260620 18:38:46.730748 3629 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:38:46 dist-test-slave-5lgp krb5kdc[3616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980726, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:46.737743 3629 init.cc:377] Logged in from keytab as kudu/127.22.37.62@KRBTEST.COM (short username kudu)
I20260620 18:38:46.739223 3629 webserver.cc:460] Webserver started at http://127.22.37.62:44327/ using document root <none> and password file <none>
I20260620 18:38:46.739924 3629 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:46.740067 3629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:46.740401 3629 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:46.743432 3629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980527431848-22676-0/minicluster-data/master-0/data/instance:
uuid: "3c0eff881981446f97adff33bf21f9c0"
format_stamp: "Formatted at 2026-06-20 18:38:46 on dist-test-slave-5lgp"
server_key: "08e62800d4fa4878e3953bcb808b5e16"
server_key_iv: "46fce0088a6b09230a071cf46e48cb3a"
server_key_version: "encryptionkey@0"
I20260620 18:38:46.744218 3629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980527431848-22676-0/minicluster-data/master-0/wal/instance:
uuid: "3c0eff881981446f97adff33bf21f9c0"
format_stamp: "Formatted at 2026-06-20 18:38:46 on dist-test-slave-5lgp"
server_key: "08e62800d4fa4878e3953bcb808b5e16"
server_key_iv: "46fce0088a6b09230a071cf46e48cb3a"
server_key_version: "encryptionkey@0"
I20260620 18:38:46.749313 3629 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 18:38:46.752558 3644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:46.753995 3629 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:38:46.754194 3629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980527431848-22676-0/minicluster-data/master-0/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980527431848-22676-0/minicluster-data/master-0/wal
uuid: "3c0eff881981446f97adff33bf21f9c0"
format_stamp: "Formatted at 2026-06-20 18:38:46 on dist-test-slave-5lgp"
server_key: "08e62800d4fa4878e3953bcb808b5e16"
server_key_iv: "46fce0088a6b09230a071cf46e48cb3a"
server_key_version: "encryptionkey@0"
I20260620 18:38:46.754424 3629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980527431848-22676-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980527431848-22676-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980527431848-22676-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:46.782929 3629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:46.786875 3629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:46.787171 3629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:46.805413 3629 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.62:40755
I20260620 18:38:46.805505 3695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.62:40755 every 8 connection(s)
I20260620 18:38:46.806825 3629 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
I20260620 18:38:46.810432 3696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:46.816048 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 3629
I20260620 18:38:46.816263 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980527431848-22676-0/minicluster-data/master-0/wal/instance
I20260620 18:38:46.816917 22676 external_mini_cluster.cc:1428] Setting key 22cc022afed06252c9bf11e1aaa1743c
I20260620 18:38:46.817222 3696 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.37.62" port: 40755 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:38:46 dist-test-slave-5lgp krb5kdc[3616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980726, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.62@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.37.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.37.61@KRBTEST.COM" created.
I20260620 18:38:46.839665 3699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:46.821921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42336 (local address 127.22.37.62:40755)
0620 18:38:46.822635 (+ 714us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:46.822647 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:46.822702 (+ 55us) server_negotiation.cc:407] Connection header received
0620 18:38:46.823903 (+ 1201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:46.823909 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:46.824140 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:46.824347 (+ 207us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:46.825661 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:46.826846 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:46.828225 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:46.828772 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:46.828970 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:46.832527 (+ 3557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:46.832535 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:46.832547 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:46.832578 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:46.835317 (+ 2739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:46.835979 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:46.835987 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:46.835989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:46.836128 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:46.836523 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:46.836527 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:46.836529 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:46.837004 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:46.837209 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:46.837829 (+ 620us) server_negotiation.cc:299] Negotiation successful
0620 18:38:46.838204 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":516,"thread_start_us":336,"threads_started":1}
I20260620 18:38:46.839686 3698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:46.820486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:40755 (local address 127.0.0.1:42336)
0620 18:38:46.821441 (+ 955us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:46.821488 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:46.823537 (+ 2049us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:46.824357 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:46.824363 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:46.825063 (+ 700us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:46.825430 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:46.825448 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:46.827008 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:46.827013 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:46.828090 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:46.828102 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:46.828989 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:46.828996 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:46.829185 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:46.829897 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:46.829919 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:46.832312 (+ 2393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:46.835446 (+ 3134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:46.835458 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:46.835470 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:46.835845 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:46.836231 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:46.836238 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:46.836241 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:46.836414 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:46.837145 (+ 731us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:46.837152 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:46.837360 (+ 208us) client_negotiation.cc:769] Sending connection context
0620 18:38:46.837681 (+ 321us) client_negotiation.cc:240] Negotiation successful
0620 18:38:46.838312 (+ 631us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":667,"mutex_wait_us":153,"thread_start_us":401,"threads_started":1}
I20260620 18:38:46.843065 3696 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.37.61" port: 42843 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:38:46.844226 3698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:46.843843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.61:42843 (local address 127.0.0.1:59154)
0620 18:38:46.844009 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:46.844115 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.61:42843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 18:38:46.844568 3648 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.37.61:42843: connect: Connection refused (error 111)
I20260620 18:38:46.846334 3698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:46.845887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.61:42843 (local address 127.0.0.1:59160)
0620 18:38:46.846068 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:46.846153 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.61:42843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 18:38:46.846647 3696 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.37.61:42843: Network error: Client connection negotiation failed: client connection to 127.22.37.61:42843: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.37.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.61.keytab.
Entry for principal kudu/127.22.37.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.61.keytab.
I20260620 18:38:46.866889 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980527431848-22676-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980527431848-22676-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980527431848-22676-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980527431848-22676-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.61:42843
--webserver_interface=127.22.37.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.61.keytab
--principal=kudu/127.22.37.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:34989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.37.62:40755,127.22.37.61:42843,127.22.37.60:35949
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:38:46.895998 3696 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.37.61" port: 42843 } attempt: 1
I20260620 18:38:46.897164 3698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:46.896751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.61:42843 (local address 127.0.0.1:59168)
0620 18:38:46.896938 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:46.897025 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.61:42843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260620 18:38:46.898124 3698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:46.897809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.61:42843 (local address 127.0.0.1:59172)
0620 18:38:46.897943 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:46.898010 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.61:42843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 18:38:46.898303 3696 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.37.61:42843: Network error: Client connection negotiation failed: client connection to 127.22.37.61:42843: connect: Connection refused (error 111)
I20260620 18:38:46.966574 3696 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.37.61" port: 42843 } attempt: 2
I20260620 18:38:46.967774 3698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:46.967287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.61:42843 (local address 127.0.0.1:59180)
0620 18:38:46.967541 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:46.967629 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.61:42843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260620 18:38:46.968699 3698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:46.968367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.61:42843 (local address 127.0.0.1:59196)
0620 18:38:46.968507 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:46.968583 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.61:42843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 18:38:46.968873 3696 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.37.61:42843: Network error: Client connection negotiation failed: client connection to 127.22.37.61:42843: connect: Connection refused (error 111)
I20260620 18:38:47.060096 3696 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.37.61" port: 42843 } attempt: 3
I20260620 18:38:47.061209 3698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:47.060756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.61:42843 (local address 127.0.0.1:59200)
0620 18:38:47.060953 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:47.061077 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.61:42843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260620 18:38:47.062176 3698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:47.061844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.61:42843 (local address 127.0.0.1:59216)
0620 18:38:47.061979 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:47.062052 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.61:42843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260620 18:38:47.062331 3696 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.37.61:42843: Network error: Client connection negotiation failed: client connection to 127.22.37.61:42843: connect: Connection refused (error 111)
W20260620 18:38:47.199627 3703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:47.199958 3703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:47.200126 3703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 18:38:47.205556 3696 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.37.61" port: 42843 } attempt: 4
I20260620 18:38:47.206897 3698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:47.206197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.61:42843 (local address 127.0.0.1:59218)
0620 18:38:47.206558 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:47.206701 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.61:42843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160}
I20260620 18:38:47.207985 3698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:47.207568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.61:42843 (local address 127.0.0.1:59226)
0620 18:38:47.207717 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:47.207835 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.61:42843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 18:38:47.208184 3696 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.37.61:42843: Network error: Client connection negotiation failed: client connection to 127.22.37.61:42843: connect: Connection refused (error 111)
W20260620 18:38:47.208282 3703 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:38:47.208403 3703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:47.208519 3703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:47.208624 3703 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:38:47.208696 3703 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:38:47.218881 3703 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:34989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980527431848-22676-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980527431848-22676-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.22.37.62:40755,127.22.37.61:42843,127.22.37.60:35949
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.61.keytab
--principal=kudu/127.22.37.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.61:42843
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980527431848-22676-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.22.37.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980527431848-22676-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:38:47.220322 3703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:47.221753 3703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:47.233724 3711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:47.233866 3708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:47.234069 3709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:47.234809 3703 server_base.cc:1034] running on GCE node
I20260620 18:38:47.235597 3703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:47.236930 3703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:47.238152 3703 hybrid_clock.cc:648] HybridClock initialized: now 1781980727238067 us; error 75 us; skew 500 ppm
I20260620 18:38:47.238483 3703 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:38:47 dist-test-slave-5lgp krb5kdc[3616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980727, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:47.241654 3703 init.cc:377] Logged in from keytab as kudu/127.22.37.61@KRBTEST.COM (short username kudu)
I20260620 18:38:47.243028 3703 webserver.cc:460] Webserver started at http://127.22.37.61:38131/ using document root <none> and password file <none>
I20260620 18:38:47.243803 3703 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:47.243957 3703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:47.244306 3703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:47.247743 3703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980527431848-22676-0/minicluster-data/master-1/data/instance:
uuid: "3a40111054f943afae3613a638287a16"
format_stamp: "Formatted at 2026-06-20 18:38:47 on dist-test-slave-5lgp"
server_key: "60d9eebd4610d8eb936dc76c93741731"
server_key_iv: "d055fd5d0a523779c1082ecdeb04fd7e"
server_key_version: "encryptionkey@0"
I20260620 18:38:47.248608 3703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980527431848-22676-0/minicluster-data/master-1/wal/instance:
uuid: "3a40111054f943afae3613a638287a16"
format_stamp: "Formatted at 2026-06-20 18:38:47 on dist-test-slave-5lgp"
server_key: "60d9eebd4610d8eb936dc76c93741731"
server_key_iv: "d055fd5d0a523779c1082ecdeb04fd7e"
server_key_version: "encryptionkey@0"
I20260620 18:38:47.254019 3703 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 18:38:47.257570 3718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:47.259012 3703 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:38:47.259217 3703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980527431848-22676-0/minicluster-data/master-1/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980527431848-22676-0/minicluster-data/master-1/wal
uuid: "3a40111054f943afae3613a638287a16"
format_stamp: "Formatted at 2026-06-20 18:38:47 on dist-test-slave-5lgp"
server_key: "60d9eebd4610d8eb936dc76c93741731"
server_key_iv: "d055fd5d0a523779c1082ecdeb04fd7e"
server_key_version: "encryptionkey@0"
I20260620 18:38:47.259430 3703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980527431848-22676-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980527431848-22676-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980527431848-22676-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:47.280107 3703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:47.284636 3703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:47.284942 3703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:47.302874 3703 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.61:42843
I20260620 18:38:47.302897 3769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.61:42843 every 8 connection(s)
I20260620 18:38:47.304260 3703 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980527431848-22676-0/minicluster-data/master-1/data/info.pb
I20260620 18:38:47.308085 3770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:47.308676 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 3703
I20260620 18:38:47.308861 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980527431848-22676-0/minicluster-data/master-1/wal/instance
I20260620 18:38:47.309482 22676 external_mini_cluster.cc:1428] Setting key 4af3c4976c3af2c1b947ed46b95e3d1b
I20260620 18:38:47.318200 3770 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.37.62" port: 40755 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.22.37.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.37.60@KRBTEST.COM" created.
Jun 20 18:38:47 dist-test-slave-5lgp krb5kdc[3616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980727, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.61@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM
I20260620 18:38:47.342377 3699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:47.320865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42350 (local address 127.22.37.62:40755)
0620 18:38:47.321102 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:47.321110 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:47.322552 (+ 1442us) server_negotiation.cc:407] Connection header received
0620 18:38:47.324406 (+ 1854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:47.324412 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:47.324511 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:47.324626 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:47.326233 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:47.326923 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:47.328494 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:47.328920 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:47.329065 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:47.335145 (+ 6080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:47.335154 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:47.335162 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:47.335216 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:47.339043 (+ 3827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:47.339828 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:47.339838 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:47.339845 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:47.340010 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:47.340437 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:47.340445 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:47.340451 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:47.340873 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:47.341063 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:47.341798 (+ 735us) server_negotiation.cc:299] Negotiation successful
0620 18:38:47.342046 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 18:38:47.343776 3773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:47.321557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:40755 (local address 127.0.0.1:42350)
0620 18:38:47.322372 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:47.322411 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:47.324129 (+ 1718us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:47.324869 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:47.324877 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:47.325463 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:47.326033 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:47.326050 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:47.327099 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:47.327103 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:47.328325 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:47.328338 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:47.329173 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:47.329179 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:47.329468 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:47.330194 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:47.330217 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:47.334867 (+ 4650us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:47.339225 (+ 4358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:47.339232 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:47.339247 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:47.339681 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:47.340120 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:47.340123 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:47.340126 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:47.340318 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:47.341008 (+ 690us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:47.341014 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:47.341358 (+ 344us) client_negotiation.cc:769] Sending connection context
0620 18:38:47.341692 (+ 334us) client_negotiation.cc:240] Negotiation successful
0620 18:38:47.342240 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":367,"threads_started":1}
I20260620 18:38:47.345672 3770 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.37.61" port: 42843 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:38:47 dist-test-slave-5lgp krb5kdc[3616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980727, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.61@KRBTEST.COM for kudu/127.22.37.61@KRBTEST.COM
I20260620 18:38:47.362118 3773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:47.346491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.61:42843 (local address 127.0.0.1:59236)
0620 18:38:47.346725 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:47.346746 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:47.346857 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:47.347846 (+ 989us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:47.347854 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:47.348386 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:47.348620 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:47.348635 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:47.350079 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:47.350087 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:47.351707 (+ 1620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:47.351723 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:47.352588 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:47.352600 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:47.352810 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:47.353501 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:47.353529 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:47.356110 (+ 2581us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:47.359736 (+ 3626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:47.359740 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:47.359742 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:47.360077 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:47.360514 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:47.360518 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:47.360520 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:47.360658 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:47.361319 (+ 661us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:47.361322 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:47.361502 (+ 180us) client_negotiation.cc:769] Sending connection context
0620 18:38:47.361617 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 18:38:47.361815 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260620 18:38:47.362597 3775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:47.346682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59236 (local address 127.22.37.61:42843)
0620 18:38:47.347361 (+ 679us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:47.347367 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:47.347393 (+ 26us) server_negotiation.cc:407] Connection header received
0620 18:38:47.347471 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:47.347478 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:47.347688 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:47.347815 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:47.348787 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:47.349946 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:47.351867 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:47.352434 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:47.352539 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:47.356319 (+ 3780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:47.356330 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:47.356344 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:47.356375 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:47.359624 (+ 3249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:47.360184 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:47.360190 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:47.360192 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:47.360383 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:47.360765 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:47.360768 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:47.360770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:47.361142 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:47.361449 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:47.361830 (+ 381us) server_negotiation.cc:299] Negotiation successful
0620 18:38:47.362046 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":378,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.37.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.60.keytab.
Entry for principal kudu/127.22.37.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.60.keytab.
I20260620 18:38:47.364904 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980527431848-22676-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980527431848-22676-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980527431848-22676-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980527431848-22676-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.60:35949
--webserver_interface=127.22.37.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.60.keytab
--principal=kudu/127.22.37.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:34989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.37.62:40755,127.22.37.61:42843,127.22.37.60:35949
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:38:47.365296 3770 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.37.60" port: 35949 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:38:47.366428 3773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:47.365968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.60:35949 (local address 127.0.0.1:43836)
0620 18:38:47.366153 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:47.366288 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.60:35949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260620 18:38:47.366660 3719 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.37.60:35949: connect: Connection refused (error 111)
I20260620 18:38:47.368214 3773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:47.367887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.60:35949 (local address 127.0.0.1:43848)
0620 18:38:47.368039 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:47.368120 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.60:35949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 18:38:47.368402 3770 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.37.60:35949: Network error: Client connection negotiation failed: client connection to 127.22.37.60:35949: connect: Connection refused (error 111)
I20260620 18:38:47.417721 3770 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.37.60" port: 35949 } attempt: 1
I20260620 18:38:47.419025 3773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:47.418439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.60:35949 (local address 127.0.0.1:43856)
0620 18:38:47.418676 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:47.418817 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.60:35949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260620 18:38:47.420053 3773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:47.419669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.60:35949 (local address 127.0.0.1:43872)
0620 18:38:47.419814 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:47.419923 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.60:35949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 18:38:47.420329 3770 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.37.60:35949: Network error: Client connection negotiation failed: client connection to 127.22.37.60:35949: connect: Connection refused (error 111)
I20260620 18:38:47.488588 3770 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.37.60" port: 35949 } attempt: 2
I20260620 18:38:47.489841 3773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:47.489313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.60:35949 (local address 127.0.0.1:43888)
0620 18:38:47.489579 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:47.489676 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.60:35949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260620 18:38:47.490916 3773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:47.490578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.60:35949 (local address 127.0.0.1:43890)
0620 18:38:47.490732 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:47.490796 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.60:35949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 18:38:47.491166 3770 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.37.60:35949: Network error: Client connection negotiation failed: client connection to 127.22.37.60:35949: connect: Connection refused (error 111)
I20260620 18:38:47.507450 3696 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.37.61" port: 42843 } attempt: 5
Jun 20 18:38:47 dist-test-slave-5lgp krb5kdc[3616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980726, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.62@KRBTEST.COM for kudu/127.22.37.61@KRBTEST.COM
I20260620 18:38:47.520474 3775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:47.508300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59248 (local address 127.22.37.61:42843)
0620 18:38:47.508497 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:47.508504 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:47.508626 (+ 122us) server_negotiation.cc:407] Connection header received
0620 18:38:47.508835 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:47.508842 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:47.508924 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:47.509032 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:47.509882 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:47.510578 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:47.511731 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:47.512197 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:47.512339 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:47.515338 (+ 2999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:47.515343 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:47.515346 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:47.515379 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:47.517988 (+ 2609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:47.518610 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:47.518618 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:47.518622 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:47.518746 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:47.519112 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:47.519118 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:47.519121 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:47.519403 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:47.519557 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:47.519917 (+ 360us) server_negotiation.cc:299] Negotiation successful
0620 18:38:47.520127 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260620 18:38:47.520471 3698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:47.508274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.61:42843 (local address 127.0.0.1:59248)
0620 18:38:47.508496 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:47.508517 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:47.508707 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:47.509067 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:47.509072 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:47.509548 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:47.509725 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:47.509733 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:47.510709 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:47.510716 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:47.511594 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:47.511604 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:47.512346 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:47.512352 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:47.512534 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:47.513148 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:47.513165 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:47.515143 (+ 1978us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:47.518120 (+ 2977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:47.518126 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:47.518130 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:47.518497 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:47.518862 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:47.518867 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:47.518871 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:47.519005 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:47.519519 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:47.519525 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:47.519676 (+ 151us) client_negotiation.cc:769] Sending connection context
0620 18:38:47.519817 (+ 141us) client_negotiation.cc:240] Negotiation successful
0620 18:38:47.520102 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79}
I20260620 18:38:47.521806 3696 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.37.60" port: 35949 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:38:47.522902 3698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:47.522490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.60:35949 (local address 127.0.0.1:43896)
0620 18:38:47.522656 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:47.522741 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.60:35949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260620 18:38:47.523788 3698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:47.523491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.60:35949 (local address 127.0.0.1:43902)
0620 18:38:47.523641 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:47.523701 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.60:35949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 18:38:47.524001 3696 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.37.60:35949: Network error: Client connection negotiation failed: client connection to 127.22.37.60:35949: connect: Connection refused (error 111)
I20260620 18:38:47.575238 3696 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.37.60" port: 35949 } attempt: 1
I20260620 18:38:47.576622 3698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:47.575938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.60:35949 (local address 127.0.0.1:43904)
0620 18:38:47.576219 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:47.576377 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.60:35949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260620 18:38:47.577754 3698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:47.577411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.60:35949 (local address 127.0.0.1:43912)
0620 18:38:47.577540 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:47.577625 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.60:35949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260620 18:38:47.577968 3696 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.37.60:35949: Network error: Client connection negotiation failed: client connection to 127.22.37.60:35949: connect: Connection refused (error 111)
I20260620 18:38:47.582434 3770 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.37.60" port: 35949 } attempt: 3
I20260620 18:38:47.583527 3773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:47.583068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.60:35949 (local address 127.0.0.1:43924)
0620 18:38:47.583292 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:47.583383 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.60:35949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260620 18:38:47.584578 3773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:47.584167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.60:35949 (local address 127.0.0.1:43928)
0620 18:38:47.584379 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:47.584466 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.60:35949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
W20260620 18:38:47.584817 3770 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.37.60:35949: Network error: Client connection negotiation failed: client connection to 127.22.37.60:35949: connect: Connection refused (error 111)
I20260620 18:38:47.646186 3696 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.37.60" port: 35949 } attempt: 2
I20260620 18:38:47.647476 3698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:47.646892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.60:35949 (local address 127.0.0.1:43930)
0620 18:38:47.647139 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:47.647261 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.60:35949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260620 18:38:47.648752 3698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:47.648337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.60:35949 (local address 127.0.0.1:43944)
0620 18:38:47.648527 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:47.648613 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.60:35949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260620 18:38:47.649050 3696 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.37.60:35949: Network error: Client connection negotiation failed: client connection to 127.22.37.60:35949: connect: Connection refused (error 111)
W20260620 18:38:47.700862 3777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:47.701195 3777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:47.701385 3777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:47.708618 3777 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:38:47.708720 3777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:47.708827 3777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:47.708910 3777 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:38:47.708983 3777 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:38:47.718896 3777 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:34989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980527431848-22676-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980527431848-22676-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.22.37.62:40755,127.22.37.61:42843,127.22.37.60:35949
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.60.keytab
--principal=kudu/127.22.37.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.60:35949
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980527431848-22676-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.22.37.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980527431848-22676-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:38:47.720239 3777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:47.721627 3777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 18:38:47.728047 3770 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.37.60" port: 35949 } attempt: 4
I20260620 18:38:47.729162 3773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:47.728709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.60:35949 (local address 127.0.0.1:43952)
0620 18:38:47.728922 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:47.729010 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.60:35949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260620 18:38:47.730136 3773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:47.729814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.60:35949 (local address 127.0.0.1:43958)
0620 18:38:47.729964 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:47.730040 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.60:35949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 18:38:47.730341 3770 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.37.60:35949: Network error: Client connection negotiation failed: client connection to 127.22.37.60:35949: connect: Connection refused (error 111)
W20260620 18:38:47.734016 3783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:47.733978 3785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:47.734097 3782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:47.735074 3777 server_base.cc:1034] running on GCE node
I20260620 18:38:47.735961 3777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:47.737509 3777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:47.738709 3777 hybrid_clock.cc:648] HybridClock initialized: now 1781980727738662 us; error 50 us; skew 500 ppm
I20260620 18:38:47.739028 3777 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:38:47 dist-test-slave-5lgp krb5kdc[3616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980727, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:47.742252 3777 init.cc:377] Logged in from keytab as kudu/127.22.37.60@KRBTEST.COM (short username kudu)
I20260620 18:38:47.743697 3777 webserver.cc:460] Webserver started at http://127.22.37.60:38665/ using document root <none> and password file <none>
I20260620 18:38:47.744375 3777 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:47.744472 3777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:47.744717 3777 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:47.747807 3777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980527431848-22676-0/minicluster-data/master-2/data/instance:
uuid: "f61032d270e94b119adea63c58d414f0"
format_stamp: "Formatted at 2026-06-20 18:38:47 on dist-test-slave-5lgp"
server_key: "d1f1a3b95218a70dd815f9641ca8e4d3"
server_key_iv: "7518b4dbd2a98286045a8a5bdc4e7c3b"
server_key_version: "encryptionkey@0"
I20260620 18:38:47.748728 3777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980527431848-22676-0/minicluster-data/master-2/wal/instance:
uuid: "f61032d270e94b119adea63c58d414f0"
format_stamp: "Formatted at 2026-06-20 18:38:47 on dist-test-slave-5lgp"
server_key: "d1f1a3b95218a70dd815f9641ca8e4d3"
server_key_iv: "7518b4dbd2a98286045a8a5bdc4e7c3b"
server_key_version: "encryptionkey@0"
I20260620 18:38:47.754141 3777 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 18:38:47.755291 3696 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.37.60" port: 35949 } attempt: 3
I20260620 18:38:47.756426 3698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:47.755937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.60:35949 (local address 127.0.0.1:43964)
0620 18:38:47.756178 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:47.756272 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.60:35949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260620 18:38:47.757663 3698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:47.757221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.60:35949 (local address 127.0.0.1:43972)
0620 18:38:47.757492 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:47.757567 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.60:35949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
W20260620 18:38:47.758003 3696 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.37.60:35949: Network error: Client connection negotiation failed: client connection to 127.22.37.60:35949: connect: Connection refused (error 111)
I20260620 18:38:47.758276 3792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:47.759812 3777 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:38:47.760006 3777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980527431848-22676-0/minicluster-data/master-2/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980527431848-22676-0/minicluster-data/master-2/wal
uuid: "f61032d270e94b119adea63c58d414f0"
format_stamp: "Formatted at 2026-06-20 18:38:47 on dist-test-slave-5lgp"
server_key: "d1f1a3b95218a70dd815f9641ca8e4d3"
server_key_iv: "7518b4dbd2a98286045a8a5bdc4e7c3b"
server_key_version: "encryptionkey@0"
I20260620 18:38:47.760237 3777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980527431848-22676-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980527431848-22676-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980527431848-22676-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:47.824556 3777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:47.828617 3777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:47.828917 3777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:47.847112 3777 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.60:35949
I20260620 18:38:47.847144 3843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.60:35949 every 8 connection(s)
I20260620 18:38:47.848472 3777 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980527431848-22676-0/minicluster-data/master-2/data/info.pb
I20260620 18:38:47.852363 3844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:47.858019 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 3777
I20260620 18:38:47.858202 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980527431848-22676-0/minicluster-data/master-2/wal/instance
I20260620 18:38:47.858729 22676 external_mini_cluster.cc:1428] Setting key fbdb899378328d27f23fd34e3682cef9
I20260620 18:38:47.859823 3844 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.37.62" port: 40755 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:38:47 dist-test-slave-5lgp krb5kdc[3616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980727, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.60@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM
Jun 20 18:38:47 dist-test-slave-5lgp krb5kdc[3616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980726, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.37.60@KRBTEST.COM
I20260620 18:38:47.881701 3848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:47.862695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42366 (local address 127.22.37.62:40755)
0620 18:38:47.863346 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:47.863354 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:47.863994 (+ 640us) server_negotiation.cc:407] Connection header received
0620 18:38:47.864471 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:47.864476 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:47.864565 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:47.864703 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:47.866615 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:47.867764 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:47.869865 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:47.870476 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:47.870606 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:47.874664 (+ 4058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:47.874669 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:47.874672 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:47.874703 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:47.877685 (+ 2982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:47.878585 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:47.878592 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:47.878596 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:47.878755 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:47.879418 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:47.879423 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:47.879426 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:47.879786 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:47.879910 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:47.880869 (+ 959us) server_negotiation.cc:299] Negotiation successful
0620 18:38:47.881119 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":372,"threads_started":1}
I20260620 18:38:47.882810 3847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:47.860668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43984 (local address 127.22.37.60:35949)
0620 18:38:47.861578 (+ 910us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:47.861589 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:47.861630 (+ 41us) server_negotiation.cc:407] Connection header received
0620 18:38:47.864233 (+ 2603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:47.864251 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:47.864485 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:47.864672 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:47.866483 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:47.868139 (+ 1656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:47.870016 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:47.870885 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:47.870974 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:47.875348 (+ 4374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:47.875357 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:47.875369 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:47.875400 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:47.878180 (+ 2780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:47.879101 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:47.879109 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:47.879112 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:47.879257 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:47.879691 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:47.879695 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:47.879697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:47.880418 (+ 721us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:47.880718 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:47.881076 (+ 358us) server_negotiation.cc:299] Negotiation successful
0620 18:38:47.881441 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":174,"server-negotiator.queue_time_us":641,"spinlock_wait_cycles":12672,"thread_start_us":417,"threads_started":1}
I20260620 18:38:47.882838 3849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:47.862712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:40755 (local address 127.0.0.1:42366)
0620 18:38:47.863816 (+ 1104us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:47.863874 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:47.864238 (+ 364us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:47.864940 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:47.864948 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:47.865858 (+ 910us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:47.866434 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:47.866461 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:47.867919 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:47.867923 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:47.869718 (+ 1795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:47.869733 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:47.870684 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:47.870688 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:47.870917 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:47.871702 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:47.871736 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:47.874475 (+ 2739us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:47.877813 (+ 3338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:47.877826 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:47.877838 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:47.878442 (+ 604us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:47.878908 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:47.878912 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:47.878915 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:47.879189 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:47.879922 (+ 733us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:47.879930 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:47.880425 (+ 495us) client_negotiation.cc:769] Sending connection context
0620 18:38:47.880713 (+ 288us) client_negotiation.cc:240] Negotiation successful
0620 18:38:47.881321 (+ 608us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":875,"mutex_wait_us":101,"spinlock_wait_cycles":17152,"thread_start_us":611,"threads_started":1}
I20260620 18:38:47.884509 3844 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.37.61" port: 42843 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:38:47 dist-test-slave-5lgp krb5kdc[3616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980727, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.60@KRBTEST.COM for kudu/127.22.37.61@KRBTEST.COM
Jun 20 18:38:47 dist-test-slave-5lgp krb5kdc[3616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980726, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM
I20260620 18:38:47.898337 3775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:47.885328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59252 (local address 127.22.37.61:42843)
0620 18:38:47.885574 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:47.885582 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:47.885646 (+ 64us) server_negotiation.cc:407] Connection header received
0620 18:38:47.885816 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:47.885822 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:47.885918 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:47.886012 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:47.887063 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:47.887737 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:47.889073 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:47.889538 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:47.889631 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:47.892832 (+ 3201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:47.892837 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:47.892840 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:47.892872 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:47.895581 (+ 2709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:47.896458 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:47.896461 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:47.896464 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:47.896631 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:47.897080 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:47.897084 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:47.897086 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:47.897333 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:47.897521 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:47.897905 (+ 384us) server_negotiation.cc:299] Negotiation successful
0620 18:38:47.898069 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260620 18:38:47.898514 3849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:47.885420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.61:42843 (local address 127.0.0.1:59252)
0620 18:38:47.885566 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:47.885586 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:47.885685 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:47.886231 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:47.886236 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:47.886697 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:47.886902 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:47.886916 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:47.887904 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:47.887911 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:47.888929 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:47.888943 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:47.889671 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:47.889677 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:47.889880 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:47.890493 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:47.890510 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:47.892641 (+ 2131us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:47.895711 (+ 3070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:47.895715 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:47.895738 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:47.896298 (+ 560us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:47.896748 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:47.896773 (+ 25us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:47.896777 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:47.896962 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:47.897515 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:47.897520 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:47.897681 (+ 161us) client_negotiation.cc:769] Sending connection context
0620 18:38:47.897843 (+ 162us) client_negotiation.cc:240] Negotiation successful
0620 18:38:47.898129 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260620 18:38:47.899816 3844 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.37.60" port: 35949 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:38:47.901618 3848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:47.885895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42376 (local address 127.22.37.62:40755)
0620 18:38:47.886046 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:47.886050 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:47.886096 (+ 46us) server_negotiation.cc:407] Connection header received
0620 18:38:47.886260 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:47.886264 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:47.886329 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:47.886404 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:47.888127 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:47.888770 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:47.890375 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:47.890793 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:47.890899 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:47.895100 (+ 4201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:47.895104 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:47.895108 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:47.895138 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:47.898447 (+ 3309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:47.899329 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:47.899336 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:47.899342 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:47.899537 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:47.900090 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:47.900093 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:47.900095 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:47.900360 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:47.900649 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:47.901038 (+ 389us) server_negotiation.cc:299] Negotiation successful
0620 18:38:47.901265 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
Jun 20 18:38:47 dist-test-slave-5lgp krb5kdc[3616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980727, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.60@KRBTEST.COM for kudu/127.22.37.60@KRBTEST.COM
Jun 20 18:38:47 dist-test-slave-5lgp krb5kdc[3616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980726, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.37.61@KRBTEST.COM
I20260620 18:38:47.914495 3847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:47.900608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43990 (local address 127.22.37.60:35949)
0620 18:38:47.900772 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:47.900778 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:47.900863 (+ 85us) server_negotiation.cc:407] Connection header received
0620 18:38:47.901033 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:47.901040 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:47.901097 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:47.901195 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:47.902117 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:47.903009 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:47.904710 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:47.905214 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:47.905342 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:47.908249 (+ 2907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:47.908254 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:47.908257 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:47.908285 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:47.911566 (+ 3281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:47.912300 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:47.912306 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:47.912312 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:47.912507 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:47.912853 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:47.912858 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:47.912861 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:47.913227 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:47.913332 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:47.913813 (+ 481us) server_negotiation.cc:299] Negotiation successful
0620 18:38:47.914208 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":95,"server-negotiator.queue_time_us":59}
I20260620 18:38:47.914516 3849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:47.900567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.60:35949 (local address 127.0.0.1:43990)
0620 18:38:47.900770 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:47.900789 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:47.900924 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:47.901209 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:47.901213 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:47.901762 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:47.901943 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:47.901954 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:47.903152 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:47.903160 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:47.904552 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:47.904568 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:47.905363 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:47.905407 (+ 44us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:47.905567 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:47.906154 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:47.906170 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:47.908053 (+ 1883us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:47.911689 (+ 3636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:47.911697 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:47.911704 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:47.912197 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:47.912624 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:47.912626 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:47.912628 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:47.912746 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:47.913387 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:47.913392 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:47.913549 (+ 157us) client_negotiation.cc:769] Sending connection context
0620 18:38:47.913725 (+ 176us) client_negotiation.cc:240] Negotiation successful
0620 18:38:47.914007 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114}
I20260620 18:38:47.919075 3775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:47.903269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59264 (local address 127.22.37.61:42843)
0620 18:38:47.903423 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:47.903426 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:47.903441 (+ 15us) server_negotiation.cc:407] Connection header received
0620 18:38:47.903506 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:47.903509 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:47.903558 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:47.903629 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:47.905090 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:47.905946 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:47.907101 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:47.907510 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:47.907651 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:47.913214 (+ 5563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:47.913219 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:47.913222 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:47.913253 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:47.916495 (+ 3242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:47.917270 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:47.917273 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:47.917275 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:47.917463 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:47.917843 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:47.917846 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:47.917848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:47.918093 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:47.918283 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:47.918624 (+ 341us) server_negotiation.cc:299] Negotiation successful
0620 18:38:47.918802 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 18:38:47.920416 3844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f61032d270e94b119adea63c58d414f0: Bootstrap starting.
I20260620 18:38:47.924202 22676 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:38:47.924225 3844 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f61032d270e94b119adea63c58d414f0: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:47.925271 3844 log.cc:824] T 00000000000000000000000000000000 P f61032d270e94b119adea63c58d414f0: Log is configured to *not* fsync() on all Append() calls
I20260620 18:38:47.928614 3844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f61032d270e94b119adea63c58d414f0: No bootstrap required, opened a new log
I20260620 18:38:47.935221 3696 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.37.60" port: 35949 } attempt: 4
I20260620 18:38:47.941977 3844 raft_consensus.cc:348] T 00000000000000000000000000000000 P f61032d270e94b119adea63c58d414f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c0eff881981446f97adff33bf21f9c0" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 40755 } } peers { permanent_uuid: "3a40111054f943afae3613a638287a16" member_type: VOTER last_known_addr { host: "127.22.37.61" port: 42843 } } peers { permanent_uuid: "f61032d270e94b119adea63c58d414f0" member_type: VOTER last_known_addr { host: "127.22.37.60" port: 35949 } }
I20260620 18:38:47.943221 3844 raft_consensus.cc:374] T 00000000000000000000000000000000 P f61032d270e94b119adea63c58d414f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:47.943427 3844 raft_consensus.cc:729] T 00000000000000000000000000000000 P f61032d270e94b119adea63c58d414f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f61032d270e94b119adea63c58d414f0, State: Initialized, Role: FOLLOWER
I20260620 18:38:47.944211 3844 consensus_queue.cc:260] T 00000000000000000000000000000000 P f61032d270e94b119adea63c58d414f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c0eff881981446f97adff33bf21f9c0" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 40755 } } peers { permanent_uuid: "3a40111054f943afae3613a638287a16" member_type: VOTER last_known_addr { host: "127.22.37.61" port: 42843 } } peers { permanent_uuid: "f61032d270e94b119adea63c58d414f0" member_type: VOTER last_known_addr { host: "127.22.37.60" port: 35949 } }
Jun 20 18:38:47 dist-test-slave-5lgp krb5kdc[3616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980726, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.62@KRBTEST.COM for kudu/127.22.37.60@KRBTEST.COM
I20260620 18:38:47.961410 3859 sys_catalog.cc:455] T 00000000000000000000000000000000 P f61032d270e94b119adea63c58d414f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c0eff881981446f97adff33bf21f9c0" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 40755 } } peers { permanent_uuid: "3a40111054f943afae3613a638287a16" member_type: VOTER last_known_addr { host: "127.22.37.61" port: 42843 } } peers { permanent_uuid: "f61032d270e94b119adea63c58d414f0" member_type: VOTER last_known_addr { host: "127.22.37.60" port: 35949 } } }
I20260620 18:38:47.961956 3859 sys_catalog.cc:458] T 00000000000000000000000000000000 P f61032d270e94b119adea63c58d414f0 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:38:47.962209 3859 raft_consensus.cc:482] T 00000000000000000000000000000000 P f61032d270e94b119adea63c58d414f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:38:47.962582 3859 raft_consensus.cc:504] T 00000000000000000000000000000000 P f61032d270e94b119adea63c58d414f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c0eff881981446f97adff33bf21f9c0" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 40755 } } peers { permanent_uuid: "3a40111054f943afae3613a638287a16" member_type: VOTER last_known_addr { host: "127.22.37.61" port: 42843 } } peers { permanent_uuid: "f61032d270e94b119adea63c58d414f0" member_type: VOTER last_known_addr { host: "127.22.37.60" port: 35949 } }
I20260620 18:38:47.963174 3844 sys_catalog.cc:564] T 00000000000000000000000000000000 P f61032d270e94b119adea63c58d414f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:38:47.964800 3859 leader_election.cc:288] T 00000000000000000000000000000000 P f61032d270e94b119adea63c58d414f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c0eff881981446f97adff33bf21f9c0 (127.22.37.62:40755), 3a40111054f943afae3613a638287a16 (127.22.37.61:42843)
I20260620 18:38:47.964970 3698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:47.935908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.60:35949 (local address 127.0.0.1:44006)
0620 18:38:47.936241 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:47.936264 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:47.936402 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:47.947205 (+ 10803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:47.947212 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:47.947741 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:47.947971 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:47.947995 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:47.949777 (+ 1782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:47.949784 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:47.951316 (+ 1532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:47.951331 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:47.952371 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:47.952379 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:47.952599 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:47.953280 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:47.953305 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:47.956199 (+ 2894us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:47.961688 (+ 5489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:47.961695 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:47.961701 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:47.962231 (+ 530us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:47.962797 (+ 566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:47.962802 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:47.962808 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:47.963000 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:47.963883 (+ 883us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:47.963891 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:47.964079 (+ 188us) client_negotiation.cc:769] Sending connection context
0620 18:38:47.964220 (+ 141us) client_negotiation.cc:240] Negotiation successful
0620 18:38:47.964549 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182}
I20260620 18:38:47.968945 3775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:47.931761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59270 (local address 127.22.37.61:42843)
0620 18:38:47.931931 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:47.931937 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:47.947880 (+ 15943us) server_negotiation.cc:407] Connection header received
0620 18:38:47.948166 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:47.948173 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:47.948257 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:47.948335 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:47.950668 (+ 2333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:47.951727 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:47.954133 (+ 2406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:47.954846 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:47.954943 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:47.960095 (+ 5152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:47.960102 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:47.960106 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:47.960159 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:47.963810 (+ 3651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:47.966118 (+ 2308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:47.966124 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:47.966128 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:47.966281 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:47.966806 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:47.966815 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:47.966821 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:47.967333 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:47.967647 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:47.968083 (+ 436us) server_negotiation.cc:299] Negotiation successful
0620 18:38:47.968393 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 18:38:47.968907 3858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:47.936733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44006 (local address 127.22.37.60:35949)
0620 18:38:47.946862 (+ 10129us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:47.946870 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:47.946898 (+ 28us) server_negotiation.cc:407] Connection header received
0620 18:38:47.946955 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:47.946963 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:47.947042 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:47.947157 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:47.948194 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:47.949603 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:47.951502 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:47.952190 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:47.952942 (+ 752us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:47.956453 (+ 3511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:47.956459 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:47.956465 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:47.956505 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:47.961520 (+ 5015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:47.962504 (+ 984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:47.962509 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:47.962512 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:47.962660 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:47.963281 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:47.963288 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:47.963291 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:47.963737 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:47.965560 (+ 1823us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:47.965692 (+ 132us) server_negotiation.cc:299] Negotiation successful
0620 18:38:47.965947 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9962,"thread_start_us":9737,"threads_started":1}
I20260620 18:38:47.983100 3696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c0eff881981446f97adff33bf21f9c0: Bootstrap starting.
I20260620 18:38:47.992329 3848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:47.929134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42388 (local address 127.22.37.62:40755)
0620 18:38:47.929393 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:47.929402 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:47.943310 (+ 13908us) server_negotiation.cc:407] Connection header received
0620 18:38:47.943727 (+ 417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:47.943733 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:47.943818 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:47.943906 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:47.948960 (+ 5054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:47.950471 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:47.957615 (+ 7144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:47.958467 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:47.958608 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:47.972332 (+ 13724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:47.972342 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:47.972350 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:47.972415 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:47.978894 (+ 6479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:47.986781 (+ 7887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:47.986791 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:47.986797 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:47.987141 (+ 344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:47.990246 (+ 3105us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:47.990253 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:47.990258 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:47.990704 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:47.991004 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:47.991646 (+ 642us) server_negotiation.cc:299] Negotiation successful
0620 18:38:47.991915 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 18:38:47.992338 3696 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c0eff881981446f97adff33bf21f9c0: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:47.995214 3847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:47.932343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44002 (local address 127.22.37.60:35949)
0620 18:38:47.932562 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:47.932570 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:47.944558 (+ 11988us) server_negotiation.cc:407] Connection header received
0620 18:38:47.945080 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:47.945088 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:47.945184 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:47.945260 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:47.953182 (+ 7922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:47.954595 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:47.957429 (+ 2834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:47.958224 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:47.958367 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:47.978912 (+ 20545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:47.978922 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:47.978929 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:47.978995 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:47.991239 (+ 12244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:47.992110 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:47.992115 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:47.992120 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:47.992298 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:47.992814 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:47.992818 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:47.992821 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:47.993244 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:47.994465 (+ 1221us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:47.994624 (+ 159us) server_negotiation.cc:299] Negotiation successful
0620 18:38:47.994874 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 18:38:47.995973 3696 log.cc:824] T 00000000000000000000000000000000 P 3c0eff881981446f97adff33bf21f9c0: Log is configured to *not* fsync() on all Append() calls
I20260620 18:38:48.002844 3696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c0eff881981446f97adff33bf21f9c0: No bootstrap required, opened a new log
I20260620 18:38:48.014432 3696 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c0eff881981446f97adff33bf21f9c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c0eff881981446f97adff33bf21f9c0" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 40755 } } peers { permanent_uuid: "3a40111054f943afae3613a638287a16" member_type: VOTER last_known_addr { host: "127.22.37.61" port: 42843 } } peers { permanent_uuid: "f61032d270e94b119adea63c58d414f0" member_type: VOTER last_known_addr { host: "127.22.37.60" port: 35949 } }
I20260620 18:38:48.014940 3696 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3c0eff881981446f97adff33bf21f9c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:48.015095 3696 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c0eff881981446f97adff33bf21f9c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c0eff881981446f97adff33bf21f9c0, State: Initialized, Role: FOLLOWER
I20260620 18:38:48.015880 3696 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c0eff881981446f97adff33bf21f9c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c0eff881981446f97adff33bf21f9c0" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 40755 } } peers { permanent_uuid: "3a40111054f943afae3613a638287a16" member_type: VOTER last_known_addr { host: "127.22.37.61" port: 42843 } } peers { permanent_uuid: "f61032d270e94b119adea63c58d414f0" member_type: VOTER last_known_addr { host: "127.22.37.60" port: 35949 } }
W20260620 18:38:48.020185 3872 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f61032d270e94b119adea63c58d414f0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 18:38:48.020325 3872 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 18:38:48.020967 3874 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c0eff881981446f97adff33bf21f9c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c0eff881981446f97adff33bf21f9c0" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 40755 } } peers { permanent_uuid: "3a40111054f943afae3613a638287a16" member_type: VOTER last_known_addr { host: "127.22.37.61" port: 42843 } } peers { permanent_uuid: "f61032d270e94b119adea63c58d414f0" member_type: VOTER last_known_addr { host: "127.22.37.60" port: 35949 } } }
I20260620 18:38:48.021438 3874 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c0eff881981446f97adff33bf21f9c0 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:38:48.022387 3696 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c0eff881981446f97adff33bf21f9c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:38:48.022454 3671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f61032d270e94b119adea63c58d414f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c0eff881981446f97adff33bf21f9c0" is_pre_election: true
I20260620 18:38:48.023087 3671 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3c0eff881981446f97adff33bf21f9c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f61032d270e94b119adea63c58d414f0 in term 0.
I20260620 18:38:48.023860 3794 leader_election.cc:302] T 00000000000000000000000000000000 P f61032d270e94b119adea63c58d414f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c0eff881981446f97adff33bf21f9c0, f61032d270e94b119adea63c58d414f0; no voters:
I20260620 18:38:48.024449 3860 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f61032d270e94b119adea63c58d414f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:38:48.024613 3860 raft_consensus.cc:482] T 00000000000000000000000000000000 P f61032d270e94b119adea63c58d414f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:38:48.024753 3860 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f61032d270e94b119adea63c58d414f0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:48.028504 3860 raft_consensus.cc:504] T 00000000000000000000000000000000 P f61032d270e94b119adea63c58d414f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c0eff881981446f97adff33bf21f9c0" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 40755 } } peers { permanent_uuid: "3a40111054f943afae3613a638287a16" member_type: VOTER last_known_addr { host: "127.22.37.61" port: 42843 } } peers { permanent_uuid: "f61032d270e94b119adea63c58d414f0" member_type: VOTER last_known_addr { host: "127.22.37.60" port: 35949 } }
I20260620 18:38:48.029526 3770 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.37.60" port: 35949 } attempt: 5
I20260620 18:38:48.005604 3777 master_runner.cc:186] Error getting master registration for 127.22.37.62:40755: OK, instance_id { permanent_uuid: "3c0eff881981446f97adff33bf21f9c0" instance_seqno: 1781980726800201 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 18:38:48.030742 3777 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 18:38:48.033316 3671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f61032d270e94b119adea63c58d414f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c0eff881981446f97adff33bf21f9c0"
I20260620 18:38:48.033567 3671 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c0eff881981446f97adff33bf21f9c0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:48.036999 3671 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3c0eff881981446f97adff33bf21f9c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f61032d270e94b119adea63c58d414f0 in term 1.
I20260620 18:38:48.037457 3794 leader_election.cc:302] T 00000000000000000000000000000000 P f61032d270e94b119adea63c58d414f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c0eff881981446f97adff33bf21f9c0, f61032d270e94b119adea63c58d414f0; no voters:
I20260620 18:38:48.037792 3859 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f61032d270e94b119adea63c58d414f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:38:48.038414 3860 leader_election.cc:288] T 00000000000000000000000000000000 P f61032d270e94b119adea63c58d414f0 [CANDIDATE]: Term 1 election: Requested vote from peers 3c0eff881981446f97adff33bf21f9c0 (127.22.37.62:40755), 3a40111054f943afae3613a638287a16 (127.22.37.61:42843)
W20260620 18:38:48.040073 3650 tablet.cc:2369] T 00000000000000000000000000000000 P 3c0eff881981446f97adff33bf21f9c0: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 18:38:48 dist-test-slave-5lgp krb5kdc[3616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980727, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.61@KRBTEST.COM for kudu/127.22.37.60@KRBTEST.COM
I20260620 18:38:48.055796 3859 raft_consensus.cc:686] T 00000000000000000000000000000000 P f61032d270e94b119adea63c58d414f0 [term 1 LEADER]: Becoming Leader. State: Replica: f61032d270e94b119adea63c58d414f0, State: Running, Role: LEADER
I20260620 18:38:48.056504 3859 consensus_queue.cc:237] T 00000000000000000000000000000000 P f61032d270e94b119adea63c58d414f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c0eff881981446f97adff33bf21f9c0" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 40755 } } peers { permanent_uuid: "3a40111054f943afae3613a638287a16" member_type: VOTER last_known_addr { host: "127.22.37.61" port: 42843 } } peers { permanent_uuid: "f61032d270e94b119adea63c58d414f0" member_type: VOTER last_known_addr { host: "127.22.37.60" port: 35949 } }
I20260620 18:38:48.058822 3773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:48.030992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.60:35949 (local address 127.0.0.1:44018)
0620 18:38:48.031257 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:48.031284 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:48.031432 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:48.032608 (+ 1176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:48.032616 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:48.033323 (+ 707us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:48.033613 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:48.033630 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:48.035833 (+ 2203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:48.035841 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:48.037763 (+ 1922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:48.037780 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:48.039799 (+ 2019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:48.039809 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:48.040503 (+ 694us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:48.042004 (+ 1501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:48.042073 (+ 69us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:48.048501 (+ 6428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:48.054619 (+ 6118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:48.054629 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:48.054636 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:48.055408 (+ 772us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:48.055953 (+ 545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:48.055958 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:48.055961 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:48.056137 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:48.057615 (+ 1478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:48.057637 (+ 22us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:48.057802 (+ 165us) client_negotiation.cc:769] Sending connection context
0620 18:38:48.057941 (+ 139us) client_negotiation.cc:240] Negotiation successful
0620 18:38:48.058253 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260620 18:38:48.060230 3847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:48.031792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44018 (local address 127.22.37.60:35949)
0620 18:38:48.032025 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:48.032034 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:48.032057 (+ 23us) server_negotiation.cc:407] Connection header received
0620 18:38:48.032129 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:48.032137 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:48.032258 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:48.032380 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:48.033972 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:48.035591 (+ 1619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:48.038055 (+ 2464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:48.038989 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:48.039142 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:48.049185 (+ 10043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:48.049196 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:48.049205 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:48.049267 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:48.054414 (+ 5147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:48.055613 (+ 1199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:48.055618 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:48.055621 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:48.055803 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:48.056282 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:48.056288 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:48.056291 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:48.057475 (+ 1184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:48.059190 (+ 1715us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:48.059311 (+ 121us) server_negotiation.cc:299] Negotiation successful
0620 18:38:48.059719 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 18:38:48.067425 3770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a40111054f943afae3613a638287a16: Bootstrap starting.
I20260620 18:38:48.060806 3629 master_runner.cc:186] Error getting master registration for 127.22.37.61:42843: OK, instance_id { permanent_uuid: "3a40111054f943afae3613a638287a16" instance_seqno: 1781980727297613 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 18:38:48.071754 3629 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 18:38:48.073207 3860 sys_catalog.cc:455] T 00000000000000000000000000000000 P f61032d270e94b119adea63c58d414f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f61032d270e94b119adea63c58d414f0. Latest consensus state: current_term: 1 leader_uuid: "f61032d270e94b119adea63c58d414f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c0eff881981446f97adff33bf21f9c0" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 40755 } } peers { permanent_uuid: "3a40111054f943afae3613a638287a16" member_type: VOTER last_known_addr { host: "127.22.37.61" port: 42843 } } peers { permanent_uuid: "f61032d270e94b119adea63c58d414f0" member_type: VOTER last_known_addr { host: "127.22.37.60" port: 35949 } } }
I20260620 18:38:48.073596 3860 sys_catalog.cc:458] T 00000000000000000000000000000000 P f61032d270e94b119adea63c58d414f0 [sys.catalog]: This master's current role is: LEADER
I20260620 18:38:48.075647 3770 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a40111054f943afae3613a638287a16: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:48.077035 3770 log.cc:824] T 00000000000000000000000000000000 P 3a40111054f943afae3613a638287a16: Log is configured to *not* fsync() on all Append() calls
W20260620 18:38:48.081025 3798 tablet.cc:2369] T 00000000000000000000000000000000 P f61032d270e94b119adea63c58d414f0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:38:48.081293 3770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a40111054f943afae3613a638287a16: No bootstrap required, opened a new log
I20260620 18:38:48.086589 3888 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:38:48.088773 3888 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260620 18:38:48.091676 3886 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3c0eff881981446f97adff33bf21f9c0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 18:38:48.091809 3886 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 18:38:48.098495 3671 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3c0eff881981446f97adff33bf21f9c0 [term 1 FOLLOWER]: Refusing update from remote peer f61032d270e94b119adea63c58d414f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:38:48.099377 3860 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f61032d270e94b119adea63c58d414f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0eff881981446f97adff33bf21f9c0" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 40755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:38:48.102625 3770 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a40111054f943afae3613a638287a16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c0eff881981446f97adff33bf21f9c0" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 40755 } } peers { permanent_uuid: "3a40111054f943afae3613a638287a16" member_type: VOTER last_known_addr { host: "127.22.37.61" port: 42843 } } peers { permanent_uuid: "f61032d270e94b119adea63c58d414f0" member_type: VOTER last_known_addr { host: "127.22.37.60" port: 35949 } }
I20260620 18:38:48.103049 3770 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3a40111054f943afae3613a638287a16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:48.103163 3770 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a40111054f943afae3613a638287a16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a40111054f943afae3613a638287a16, State: Initialized, Role: FOLLOWER
I20260620 18:38:48.103804 3770 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a40111054f943afae3613a638287a16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c0eff881981446f97adff33bf21f9c0" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 40755 } } peers { permanent_uuid: "3a40111054f943afae3613a638287a16" member_type: VOTER last_known_addr { host: "127.22.37.61" port: 42843 } } peers { permanent_uuid: "f61032d270e94b119adea63c58d414f0" member_type: VOTER last_known_addr { host: "127.22.37.60" port: 35949 } }
I20260620 18:38:48.107712 3892 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a40111054f943afae3613a638287a16 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c0eff881981446f97adff33bf21f9c0" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 40755 } } peers { permanent_uuid: "3a40111054f943afae3613a638287a16" member_type: VOTER last_known_addr { host: "127.22.37.61" port: 42843 } } peers { permanent_uuid: "f61032d270e94b119adea63c58d414f0" member_type: VOTER last_known_addr { host: "127.22.37.60" port: 35949 } } }
I20260620 18:38:48.108116 3892 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a40111054f943afae3613a638287a16 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:38:48.108989 3770 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a40111054f943afae3613a638287a16 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:38:48.109339 3743 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a40111054f943afae3613a638287a16 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:48.113037 3744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f61032d270e94b119adea63c58d414f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a40111054f943afae3613a638287a16"
I20260620 18:38:48.113862 3745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f61032d270e94b119adea63c58d414f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a40111054f943afae3613a638287a16" is_pre_election: true
I20260620 18:38:48.114775 3743 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3a40111054f943afae3613a638287a16 [term 1 FOLLOWER]: Refusing update from remote peer f61032d270e94b119adea63c58d414f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:38:48.115677 3860 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f61032d270e94b119adea63c58d414f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a40111054f943afae3613a638287a16" member_type: VOTER last_known_addr { host: "127.22.37.61" port: 42843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:38:48.141809 3874 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c0eff881981446f97adff33bf21f9c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f61032d270e94b119adea63c58d414f0. Latest consensus state: current_term: 1 leader_uuid: "f61032d270e94b119adea63c58d414f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c0eff881981446f97adff33bf21f9c0" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 40755 } } peers { permanent_uuid: "3a40111054f943afae3613a638287a16" member_type: VOTER last_known_addr { host: "127.22.37.61" port: 42843 } } peers { permanent_uuid: "f61032d270e94b119adea63c58d414f0" member_type: VOTER last_known_addr { host: "127.22.37.60" port: 35949 } } }
I20260620 18:38:48.142156 3874 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c0eff881981446f97adff33bf21f9c0 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:38:48.143932 3892 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a40111054f943afae3613a638287a16 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f61032d270e94b119adea63c58d414f0. Latest consensus state: current_term: 1 leader_uuid: "f61032d270e94b119adea63c58d414f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c0eff881981446f97adff33bf21f9c0" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 40755 } } peers { permanent_uuid: "3a40111054f943afae3613a638287a16" member_type: VOTER last_known_addr { host: "127.22.37.61" port: 42843 } } peers { permanent_uuid: "f61032d270e94b119adea63c58d414f0" member_type: VOTER last_known_addr { host: "127.22.37.60" port: 35949 } } }
I20260620 18:38:48.144259 3892 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a40111054f943afae3613a638287a16 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:38:48.147491 3874 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c0eff881981446f97adff33bf21f9c0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f61032d270e94b119adea63c58d414f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c0eff881981446f97adff33bf21f9c0" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 40755 } } peers { permanent_uuid: "3a40111054f943afae3613a638287a16" member_type: VOTER last_known_addr { host: "127.22.37.61" port: 42843 } } peers { permanent_uuid: "f61032d270e94b119adea63c58d414f0" member_type: VOTER last_known_addr { host: "127.22.37.60" port: 35949 } } }
I20260620 18:38:48.147786 3874 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c0eff881981446f97adff33bf21f9c0 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:38:48.157994 3895 mvcc.cc:205] Tried to move back new op lower bound from 7298993062280613888 to 7298993062125551616. Current Snapshot: MvccSnapshot[applied={T|T < 7298993062280613888}]
I20260620 18:38:48.163035 3892 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a40111054f943afae3613a638287a16 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f61032d270e94b119adea63c58d414f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c0eff881981446f97adff33bf21f9c0" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 40755 } } peers { permanent_uuid: "3a40111054f943afae3613a638287a16" member_type: VOTER last_known_addr { host: "127.22.37.61" port: 42843 } } peers { permanent_uuid: "f61032d270e94b119adea63c58d414f0" member_type: VOTER last_known_addr { host: "127.22.37.60" port: 35949 } } }
I20260620 18:38:48.163295 3892 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a40111054f943afae3613a638287a16 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:38:48.164523 3860 sys_catalog.cc:455] T 00000000000000000000000000000000 P f61032d270e94b119adea63c58d414f0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f61032d270e94b119adea63c58d414f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c0eff881981446f97adff33bf21f9c0" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 40755 } } peers { permanent_uuid: "3a40111054f943afae3613a638287a16" member_type: VOTER last_known_addr { host: "127.22.37.61" port: 42843 } } peers { permanent_uuid: "f61032d270e94b119adea63c58d414f0" member_type: VOTER last_known_addr { host: "127.22.37.60" port: 35949 } } }
I20260620 18:38:48.164819 3860 sys_catalog.cc:458] T 00000000000000000000000000000000 P f61032d270e94b119adea63c58d414f0 [sys.catalog]: This master's current role is: LEADER
I20260620 18:38:48.165184 3888 catalog_manager.cc:1347] Generated new cluster ID: 695b9e452d6e4c56bbcca8284de2a9fc
I20260620 18:38:48.165278 3888 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:38:48.165673 3859 sys_catalog.cc:455] T 00000000000000000000000000000000 P f61032d270e94b119adea63c58d414f0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f61032d270e94b119adea63c58d414f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c0eff881981446f97adff33bf21f9c0" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 40755 } } peers { permanent_uuid: "3a40111054f943afae3613a638287a16" member_type: VOTER last_known_addr { host: "127.22.37.61" port: 42843 } } peers { permanent_uuid: "f61032d270e94b119adea63c58d414f0" member_type: VOTER last_known_addr { host: "127.22.37.60" port: 35949 } } }
I20260620 18:38:48.165964 3859 sys_catalog.cc:458] T 00000000000000000000000000000000 P f61032d270e94b119adea63c58d414f0 [sys.catalog]: This master's current role is: LEADER
I20260620 18:38:48.192114 3888 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:38:48.194099 3888 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:38:48.217545 3888 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f61032d270e94b119adea63c58d414f0: Generated new TSK 0
I20260620 18:38:48.218819 3888 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:38:48.240957 3910 catalog_manager.cc:1259] Loaded cluster ID: 695b9e452d6e4c56bbcca8284de2a9fc
I20260620 18:38:48.241170 3910 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3a40111054f943afae3613a638287a16: loading cluster ID for follower catalog manager: success
I20260620 18:38:48.244892 3910 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3a40111054f943afae3613a638287a16: acquiring CA information for follower catalog manager: success
I20260620 18:38:48.247439 3910 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3a40111054f943afae3613a638287a16: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 18:38:49.093906 3886 catalog_manager.cc:1259] Loaded cluster ID: 695b9e452d6e4c56bbcca8284de2a9fc
I20260620 18:38:49.094013 3886 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3c0eff881981446f97adff33bf21f9c0: loading cluster ID for follower catalog manager: success
I20260620 18:38:49.096683 3886 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3c0eff881981446f97adff33bf21f9c0: acquiring CA information for follower catalog manager: success
I20260620 18:38:49.098482 3886 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3c0eff881981446f97adff33bf21f9c0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 18:38:49.382541 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 3629
I20260620 18:38:49.384006 3691 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:38:49.500252 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 3629
I20260620 18:38:49.521878 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 3703
I20260620 18:38:49.522986 3765 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:38:49.632316 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 3703
I20260620 18:38:49.652803 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 3777
I20260620 18:38:49.654039 3839 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:38:49.765015 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 3777
2026-06-20T18:38:49Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4733 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980527431848-22676-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:38:49 dist-test-slave-5lgp krb5kdc[3923](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:38:49 dist-test-slave-5lgp krb5kdc[3923](info): set up 2 sockets
Jun 20 18:38:49 dist-test-slave-5lgp krb5kdc[3923](info): commencing operation
krb5kdc: starting...
W20260620 18:38:50.865918 22676 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.004s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980527431848-22676-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980527431848-22676-0/krb5kdc/test-user.keytab.
Jun 20 18:38:51 dist-test-slave-5lgp krb5kdc[3923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980731, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:38:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:38:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.37.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.37.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.37.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab.
Entry for principal kudu/127.22.37.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab.
I20260620 18:38:51.095253 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980527431848-22676-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980527431848-22676-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980527431848-22676-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.62:39151
--webserver_interface=127.22.37.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab
--principal=kudu/127.22.37.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:32991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.37.62:39151
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:38:51.431958 3936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:51.432269 3936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:51.432368 3936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:51.439785 3936 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:38:51.439899 3936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:51.439944 3936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:51.440009 3936 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:38:51.440052 3936 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:38:51.451174 3936 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:32991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980527431848-22676-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980527431848-22676-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.37.62:39151
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab
--principal=kudu/127.22.37.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.62:39151
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.37.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980527431848-22676-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:38:51.452795 3936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:51.454203 3936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:51.466391 3941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:51.466473 3942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:51.466398 3944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:51.467456 3936 server_base.cc:1034] running on GCE node
I20260620 18:38:51.468374 3936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:51.469898 3936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:51.471129 3936 hybrid_clock.cc:648] HybridClock initialized: now 1781980731471081 us; error 39 us; skew 500 ppm
I20260620 18:38:51.471483 3936 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:38:51 dist-test-slave-5lgp krb5kdc[3923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980731, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:51.474663 3936 init.cc:377] Logged in from keytab as kudu/127.22.37.62@KRBTEST.COM (short username kudu)
I20260620 18:38:51.476194 3936 webserver.cc:460] Webserver started at http://127.22.37.62:46785/ using document root <none> and password file <none>
I20260620 18:38:51.476909 3936 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:51.477058 3936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:51.477420 3936 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:51.480427 3936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980527431848-22676-0/minicluster-data/master-0/data/instance:
uuid: "d9b559d1da554c40a2762a460f64393e"
format_stamp: "Formatted at 2026-06-20 18:38:51 on dist-test-slave-5lgp"
server_key: "1f1faf91a0ea009329d0c8123171217b"
server_key_iv: "f45de5df5924e1827e1238dcbea62a6e"
server_key_version: "encryptionkey@0"
I20260620 18:38:51.481207 3936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980527431848-22676-0/minicluster-data/master-0/wal/instance:
uuid: "d9b559d1da554c40a2762a460f64393e"
format_stamp: "Formatted at 2026-06-20 18:38:51 on dist-test-slave-5lgp"
server_key: "1f1faf91a0ea009329d0c8123171217b"
server_key_iv: "f45de5df5924e1827e1238dcbea62a6e"
server_key_version: "encryptionkey@0"
I20260620 18:38:51.486441 3936 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 18:38:51.490093 3951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:51.491458 3936 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:38:51.491643 3936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980527431848-22676-0/minicluster-data/master-0/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980527431848-22676-0/minicluster-data/master-0/wal
uuid: "d9b559d1da554c40a2762a460f64393e"
format_stamp: "Formatted at 2026-06-20 18:38:51 on dist-test-slave-5lgp"
server_key: "1f1faf91a0ea009329d0c8123171217b"
server_key_iv: "f45de5df5924e1827e1238dcbea62a6e"
server_key_version: "encryptionkey@0"
I20260620 18:38:51.491856 3936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980527431848-22676-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980527431848-22676-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980527431848-22676-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:51.509665 3936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:51.513631 3936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:51.513929 3936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:51.532814 3936 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.62:39151
I20260620 18:38:51.532840 4002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.62:39151 every 8 connection(s)
I20260620 18:38:51.534363 3936 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
I20260620 18:38:51.538245 4003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:51.539149 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 3936
I20260620 18:38:51.539372 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980527431848-22676-0/minicluster-data/master-0/wal/instance
I20260620 18:38:51.540043 22676 external_mini_cluster.cc:1428] Setting key 353585bb8ac02ab903fae2381b5b0b51
I20260620 18:38:51.552912 4003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9b559d1da554c40a2762a460f64393e: Bootstrap starting.
Jun 20 18:38:51 dist-test-slave-5lgp krb5kdc[3923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980731, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM
I20260620 18:38:51.556289 4003 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d9b559d1da554c40a2762a460f64393e: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:51.557739 4003 log.cc:824] T 00000000000000000000000000000000 P d9b559d1da554c40a2762a460f64393e: Log is configured to *not* fsync() on all Append() calls
I20260620 18:38:51.561088 4003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9b559d1da554c40a2762a460f64393e: No bootstrap required, opened a new log
I20260620 18:38:51.565006 4006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:51.542076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35836 (local address 127.22.37.62:39151)
0620 18:38:51.543000 (+ 924us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:51.543016 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:51.543085 (+ 69us) server_negotiation.cc:407] Connection header received
0620 18:38:51.545095 (+ 2010us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:51.545123 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:51.545688 (+ 565us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:51.546154 (+ 466us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:51.547511 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:51.549550 (+ 2039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:51.550877 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:51.551929 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:51.552116 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:51.556425 (+ 4309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:51.556437 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:51.556457 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:51.556507 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:51.560450 (+ 3943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:51.561187 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:51.561195 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:51.561203 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:51.561436 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:51.561806 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:51.561814 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:51.561817 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:51.562527 (+ 710us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:51.562796 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:51.563183 (+ 387us) server_negotiation.cc:299] Negotiation successful
0620 18:38:51.563614 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":645,"thread_start_us":370,"threads_started":1}
I20260620 18:38:51.568938 22676 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:38:51.570716 4003 raft_consensus.cc:348] T 00000000000000000000000000000000 P d9b559d1da554c40a2762a460f64393e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9b559d1da554c40a2762a460f64393e" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 39151 } }
I20260620 18:38:51.571105 4003 raft_consensus.cc:374] T 00000000000000000000000000000000 P d9b559d1da554c40a2762a460f64393e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:51.571230 4003 raft_consensus.cc:729] T 00000000000000000000000000000000 P d9b559d1da554c40a2762a460f64393e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9b559d1da554c40a2762a460f64393e, State: Initialized, Role: FOLLOWER
I20260620 18:38:51.571902 4003 consensus_queue.cc:260] T 00000000000000000000000000000000 P d9b559d1da554c40a2762a460f64393e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9b559d1da554c40a2762a460f64393e" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 39151 } }
I20260620 18:38:51.572178 4003 raft_consensus.cc:388] T 00000000000000000000000000000000 P d9b559d1da554c40a2762a460f64393e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:38:51.572310 4003 raft_consensus.cc:482] T 00000000000000000000000000000000 P d9b559d1da554c40a2762a460f64393e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:38:51.572485 4003 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d9b559d1da554c40a2762a460f64393e [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:51.574672 4003 raft_consensus.cc:504] T 00000000000000000000000000000000 P d9b559d1da554c40a2762a460f64393e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9b559d1da554c40a2762a460f64393e" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 39151 } }
I20260620 18:38:51.575160 4003 leader_election.cc:302] T 00000000000000000000000000000000 P d9b559d1da554c40a2762a460f64393e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d9b559d1da554c40a2762a460f64393e; no voters:
I20260620 18:38:51.575742 4003 leader_election.cc:288] T 00000000000000000000000000000000 P d9b559d1da554c40a2762a460f64393e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:38:51.575906 4008 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d9b559d1da554c40a2762a460f64393e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:38:51.576558 4008 raft_consensus.cc:686] T 00000000000000000000000000000000 P d9b559d1da554c40a2762a460f64393e [term 1 LEADER]: Becoming Leader. State: Replica: d9b559d1da554c40a2762a460f64393e, State: Running, Role: LEADER
I20260620 18:38:51.577056 4008 consensus_queue.cc:237] T 00000000000000000000000000000000 P d9b559d1da554c40a2762a460f64393e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9b559d1da554c40a2762a460f64393e" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 39151 } }
I20260620 18:38:51.577827 4003 sys_catalog.cc:564] T 00000000000000000000000000000000 P d9b559d1da554c40a2762a460f64393e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:38:51.579617 4010 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9b559d1da554c40a2762a460f64393e [sys.catalog]: SysCatalogTable state changed. Reason: New leader d9b559d1da554c40a2762a460f64393e. Latest consensus state: current_term: 1 leader_uuid: "d9b559d1da554c40a2762a460f64393e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9b559d1da554c40a2762a460f64393e" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 39151 } } }
I20260620 18:38:51.579954 4010 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9b559d1da554c40a2762a460f64393e [sys.catalog]: This master's current role is: LEADER
I20260620 18:38:51.580899 4014 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:38:51.580714 4009 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9b559d1da554c40a2762a460f64393e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d9b559d1da554c40a2762a460f64393e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9b559d1da554c40a2762a460f64393e" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 39151 } } }
I20260620 18:38:51.582155 4009 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9b559d1da554c40a2762a460f64393e [sys.catalog]: This master's current role is: LEADER
I20260620 18:38:51.585477 4014 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:38:51.596310 4014 catalog_manager.cc:1347] Generated new cluster ID: 2b69c97a2ed94678b92524743bccfa7c
I20260620 18:38:51.596428 4014 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:38:51.610401 4014 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:38:51.612159 4014 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:38:51.628506 4014 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d9b559d1da554c40a2762a460f64393e: Generated new TSK 0
I20260620 18:38:51.629631 4014 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:38:51.680642 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 3936
I20260620 18:38:51.681975 3998 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:38:51.778475 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 3936
2026-06-20T18:38:51Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2007 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:38:51 dist-test-slave-5lgp krb5kdc[4031](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:38:51 dist-test-slave-5lgp krb5kdc[4031](info): set up 2 sockets
Jun 20 18:38:51 dist-test-slave-5lgp krb5kdc[4031](info): commencing operation
krb5kdc: starting...
W20260620 18:38:52.881915 22676 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/krb5kdc/test-user.keytab.
Jun 20 18:38:53 dist-test-slave-5lgp krb5kdc[4031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980733, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:38:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:38:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.37.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.37.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.37.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab.
Entry for principal kudu/127.22.37.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab.
I20260620 18:38:53.104549 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.62:37219
--webserver_interface=127.22.37.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab
--principal=kudu/127.22.37.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:36803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.37.62:37219
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:38:53.430020 4044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:53.430315 4044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:53.430501 4044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:53.437729 4044 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:38:53.437827 4044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:53.437960 4044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:53.438053 4044 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:38:53.438134 4044 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:38:53.448076 4044 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:36803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.37.62:37219
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab
--principal=kudu/127.22.37.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.62:37219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.37.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:38:53.449391 4044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:53.450706 4044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:53.463528 4052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:53.463605 4049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:53.463726 4050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:53.466828 4044 server_base.cc:1034] running on GCE node
I20260620 18:38:53.467607 4044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:53.469060 4044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:53.470283 4044 hybrid_clock.cc:648] HybridClock initialized: now 1781980733470247 us; error 51 us; skew 500 ppm
I20260620 18:38:53.470642 4044 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:38:53 dist-test-slave-5lgp krb5kdc[4031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980733, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:53.473803 4044 init.cc:377] Logged in from keytab as kudu/127.22.37.62@KRBTEST.COM (short username kudu)
I20260620 18:38:53.475405 4044 webserver.cc:460] Webserver started at http://127.22.37.62:36715/ using document root <none> and password file <none>
I20260620 18:38:53.476145 4044 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:53.476297 4044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:53.476653 4044 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:53.479796 4044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/master-0/data/instance:
uuid: "b7f7b758d94d44a19e5997546da74de4"
format_stamp: "Formatted at 2026-06-20 18:38:53 on dist-test-slave-5lgp"
server_key: "b1b4626643bf9bc61dadb969b47e777c"
server_key_iv: "d22eee2a99893023ebdce3d2b992ae36"
server_key_version: "encryptionkey@0"
I20260620 18:38:53.480578 4044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/master-0/wal/instance:
uuid: "b7f7b758d94d44a19e5997546da74de4"
format_stamp: "Formatted at 2026-06-20 18:38:53 on dist-test-slave-5lgp"
server_key: "b1b4626643bf9bc61dadb969b47e777c"
server_key_iv: "d22eee2a99893023ebdce3d2b992ae36"
server_key_version: "encryptionkey@0"
I20260620 18:38:53.485877 4044 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 18:38:53.489253 4059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:53.490561 4044 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:38:53.490744 4044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/master-0/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/master-0/wal
uuid: "b7f7b758d94d44a19e5997546da74de4"
format_stamp: "Formatted at 2026-06-20 18:38:53 on dist-test-slave-5lgp"
server_key: "b1b4626643bf9bc61dadb969b47e777c"
server_key_iv: "d22eee2a99893023ebdce3d2b992ae36"
server_key_version: "encryptionkey@0"
I20260620 18:38:53.490939 4044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:53.517547 4044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:53.521981 4044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:53.522351 4044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:53.540939 4044 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.62:37219
I20260620 18:38:53.540970 4110 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.62:37219 every 8 connection(s)
I20260620 18:38:53.542397 4044 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
I20260620 18:38:53.546000 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 4044
I20260620 18:38:53.546233 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/master-0/wal/instance
I20260620 18:38:53.546669 4111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:53.547143 22676 external_mini_cluster.cc:1428] Setting key 9b9e484c6995b1ec378793439e545d56
I20260620 18:38:53.562116 4111 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7f7b758d94d44a19e5997546da74de4: Bootstrap starting.
Jun 20 18:38:53 dist-test-slave-5lgp krb5kdc[4031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980733, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM
I20260620 18:38:53.566077 4111 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b7f7b758d94d44a19e5997546da74de4: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:53.567217 4111 log.cc:824] T 00000000000000000000000000000000 P b7f7b758d94d44a19e5997546da74de4: Log is configured to *not* fsync() on all Append() calls
I20260620 18:38:53.570204 4111 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7f7b758d94d44a19e5997546da74de4: No bootstrap required, opened a new log
I20260620 18:38:53.572057 4114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:53.549196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51476 (local address 127.22.37.62:37219)
0620 18:38:53.550176 (+ 980us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:53.550188 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:53.550232 (+ 44us) server_negotiation.cc:407] Connection header received
0620 18:38:53.551886 (+ 1654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:53.551909 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:53.552305 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:53.552637 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:53.554532 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:53.555820 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:53.557661 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:53.558360 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:53.558506 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:53.563542 (+ 5036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:53.563554 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:53.563570 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:53.563613 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:53.567300 (+ 3687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:53.568205 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:53.568214 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:53.568221 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:53.568423 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:53.568899 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:53.568905 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:53.568907 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:53.569592 (+ 685us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:53.569933 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:53.570443 (+ 510us) server_negotiation.cc:299] Negotiation successful
0620 18:38:53.570912 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":770,"thread_start_us":548,"threads_started":1}
I20260620 18:38:53.578451 4111 raft_consensus.cc:348] T 00000000000000000000000000000000 P b7f7b758d94d44a19e5997546da74de4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7f7b758d94d44a19e5997546da74de4" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 37219 } }
I20260620 18:38:53.578944 4111 raft_consensus.cc:374] T 00000000000000000000000000000000 P b7f7b758d94d44a19e5997546da74de4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:53.579056 4111 raft_consensus.cc:729] T 00000000000000000000000000000000 P b7f7b758d94d44a19e5997546da74de4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7f7b758d94d44a19e5997546da74de4, State: Initialized, Role: FOLLOWER
I20260620 18:38:53.579768 4111 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7f7b758d94d44a19e5997546da74de4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7f7b758d94d44a19e5997546da74de4" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 37219 } }
I20260620 18:38:53.580045 4111 raft_consensus.cc:388] T 00000000000000000000000000000000 P b7f7b758d94d44a19e5997546da74de4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:38:53.580194 4111 raft_consensus.cc:482] T 00000000000000000000000000000000 P b7f7b758d94d44a19e5997546da74de4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:38:53.580390 4111 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b7f7b758d94d44a19e5997546da74de4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:53.582674 4111 raft_consensus.cc:504] T 00000000000000000000000000000000 P b7f7b758d94d44a19e5997546da74de4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7f7b758d94d44a19e5997546da74de4" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 37219 } }
I20260620 18:38:53.583173 4111 leader_election.cc:302] T 00000000000000000000000000000000 P b7f7b758d94d44a19e5997546da74de4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b7f7b758d94d44a19e5997546da74de4; no voters:
I20260620 18:38:53.583839 4111 leader_election.cc:288] T 00000000000000000000000000000000 P b7f7b758d94d44a19e5997546da74de4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:38:53.584007 4116 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b7f7b758d94d44a19e5997546da74de4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:38:53.584730 4116 raft_consensus.cc:686] T 00000000000000000000000000000000 P b7f7b758d94d44a19e5997546da74de4 [term 1 LEADER]: Becoming Leader. State: Replica: b7f7b758d94d44a19e5997546da74de4, State: Running, Role: LEADER
I20260620 18:38:53.585357 4116 consensus_queue.cc:237] T 00000000000000000000000000000000 P b7f7b758d94d44a19e5997546da74de4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7f7b758d94d44a19e5997546da74de4" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 37219 } }
I20260620 18:38:53.586165 4111 sys_catalog.cc:564] T 00000000000000000000000000000000 P b7f7b758d94d44a19e5997546da74de4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:38:53.588374 4118 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7f7b758d94d44a19e5997546da74de4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b7f7b758d94d44a19e5997546da74de4. Latest consensus state: current_term: 1 leader_uuid: "b7f7b758d94d44a19e5997546da74de4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7f7b758d94d44a19e5997546da74de4" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 37219 } } }
I20260620 18:38:53.588374 4117 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7f7b758d94d44a19e5997546da74de4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b7f7b758d94d44a19e5997546da74de4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7f7b758d94d44a19e5997546da74de4" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 37219 } } }
I20260620 18:38:53.588855 4118 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7f7b758d94d44a19e5997546da74de4 [sys.catalog]: This master's current role is: LEADER
I20260620 18:38:53.588860 4117 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7f7b758d94d44a19e5997546da74de4 [sys.catalog]: This master's current role is: LEADER
I20260620 18:38:53.591425 4123 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:38:53.596702 4123 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:38:53.605796 4123 catalog_manager.cc:1347] Generated new cluster ID: 001289474cef41599d8c617f2c4b8f7b
I20260620 18:38:53.605913 4123 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:38:53.619482 4123 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:38:53.621438 4123 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:38:53.635654 4123 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b7f7b758d94d44a19e5997546da74de4: Generated new TSK 0
I20260620 18:38:53.636761 4123 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.37.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.37.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.37.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.1.keytab.
Entry for principal kudu/127.22.37.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.1.keytab.
I20260620 18:38:53.687242 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.37.1:0
--local_ip_for_outbound_sockets=127.22.37.1
--webserver_interface=127.22.37.1
--webserver_port=0
--tserver_master_addrs=127.22.37.62:37219
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.1.keytab
--principal=kudu/127.22.37.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:36803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:38:54.017880 4137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:54.018174 4137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:54.018354 4137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:54.025695 4137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:54.025811 4137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:54.026049 4137 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.37.1
I20260620 18:38:54.035939 4137 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:36803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.1.keytab
--principal=kudu/127.22.37.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.37.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.37.1
--webserver_port=0
--tserver_master_addrs=127.22.37.62:37219
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.37.1
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:38:54.037322 4137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:54.038678 4137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:54.051695 4142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:54.051757 4143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:54.051851 4145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:54.052728 4137 server_base.cc:1034] running on GCE node
I20260620 18:38:54.053414 4137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:54.054306 4137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:54.055509 4137 hybrid_clock.cc:648] HybridClock initialized: now 1781980734055469 us; error 35 us; skew 500 ppm
I20260620 18:38:54.055783 4137 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:38:54 dist-test-slave-5lgp krb5kdc[4031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980734, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:54.058880 4137 init.cc:377] Logged in from keytab as kudu/127.22.37.1@KRBTEST.COM (short username kudu)
I20260620 18:38:54.060195 4137 webserver.cc:460] Webserver started at http://127.22.37.1:41659/ using document root <none> and password file <none>
I20260620 18:38:54.060825 4137 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:54.060922 4137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:54.061175 4137 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:54.064106 4137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-0/data/instance:
uuid: "aa1a40a28b54480c80bbf29fe96919a7"
format_stamp: "Formatted at 2026-06-20 18:38:54 on dist-test-slave-5lgp"
server_key: "c42730258bd5fb3f1e73310604833a5a"
server_key_iv: "43c786ff1cc383934b071c646e1298a1"
server_key_version: "encryptionkey@0"
I20260620 18:38:54.064812 4137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-0/wal/instance:
uuid: "aa1a40a28b54480c80bbf29fe96919a7"
format_stamp: "Formatted at 2026-06-20 18:38:54 on dist-test-slave-5lgp"
server_key: "c42730258bd5fb3f1e73310604833a5a"
server_key_iv: "43c786ff1cc383934b071c646e1298a1"
server_key_version: "encryptionkey@0"
I20260620 18:38:54.069867 4137 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 18:38:54.073325 4152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:54.074713 4137 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:38:54.074846 4137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-0/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-0/wal
uuid: "aa1a40a28b54480c80bbf29fe96919a7"
format_stamp: "Formatted at 2026-06-20 18:38:54 on dist-test-slave-5lgp"
server_key: "c42730258bd5fb3f1e73310604833a5a"
server_key_iv: "43c786ff1cc383934b071c646e1298a1"
server_key_version: "encryptionkey@0"
I20260620 18:38:54.075007 4137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:54.100890 4137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:54.104777 4137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:54.105019 4137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:54.105947 4137 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:38:54.107205 4137 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:38:54.107277 4137 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:54.107360 4137 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:38:54.107405 4137 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:54.140270 4137 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.1:39491
I20260620 18:38:54.140313 4264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.1:39491 every 8 connection(s)
I20260620 18:38:54.141708 4137 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-0/data/info.pb
I20260620 18:38:54.149745 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 4137
I20260620 18:38:54.149962 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-0/wal/instance
I20260620 18:38:54.150619 22676 external_mini_cluster.cc:1428] Setting key ee0d1a0fa1ffd11534591b2c2ea91070
Jun 20 18:38:54 dist-test-slave-5lgp krb5kdc[4031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980734, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.1@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM
I20260620 18:38:54.163511 4267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:54.144259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.1:36083 (local address 127.22.37.62:37219)
0620 18:38:54.144672 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:54.144677 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:54.145702 (+ 1025us) server_negotiation.cc:407] Connection header received
0620 18:38:54.147538 (+ 1836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:54.147544 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:54.147651 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:54.147769 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:54.149262 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:54.150145 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:54.151869 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:54.152389 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:54.152518 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:54.156479 (+ 3961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:54.156489 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:54.156497 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:54.156544 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:54.160171 (+ 3627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:54.161100 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:54.161104 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:54.161106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:54.161220 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:54.161732 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:54.161735 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:54.161737 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:54.162007 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:54.162118 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:54.163030 (+ 912us) server_negotiation.cc:299] Negotiation successful
0620 18:38:54.163248 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":210,"threads_started":1}
I20260620 18:38:54.164922 4268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:54.144829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:37219 (local address 127.22.37.1:36083)
0620 18:38:54.145558 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:54.145597 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:54.147266 (+ 1669us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:54.148011 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:54.148019 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:54.148509 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:54.149060 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:54.149079 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:54.150300 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:54.150304 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:54.151715 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:54.151732 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:54.152647 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:54.152656 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:54.153013 (+ 357us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:54.153837 (+ 824us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:54.153859 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:54.156237 (+ 2378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:54.160305 (+ 4068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:54.160312 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:54.160324 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:54.160966 (+ 642us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:54.161327 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:54.161331 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:54.161335 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:54.161627 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:54.162136 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:54.162147 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:54.162590 (+ 443us) client_negotiation.cc:769] Sending connection context
0620 18:38:54.162886 (+ 296us) client_negotiation.cc:240] Negotiation successful
0620 18:38:54.163469 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":262,"threads_started":1}
I20260620 18:38:54.166520 4265 heartbeater.cc:347] Connected to a master server at 127.22.37.62:37219
I20260620 18:38:54.166818 4265 heartbeater.cc:464] Registering TS with master...
I20260620 18:38:54.167526 4265 heartbeater.cc:511] Master 127.22.37.62:37219 requested a full tablet report, sending...
I20260620 18:38:54.169592 4076 ts_manager.cc:194] Registered new tserver with Master: aa1a40a28b54480c80bbf29fe96919a7 (127.22.37.1:39491)
I20260620 18:38:54.171121 4076 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.37.1@KRBTEST.COM'} at 127.22.37.1:36083
WARNING: no policy specified for kudu/127.22.37.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.37.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.37.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.2.keytab.
Entry for principal kudu/127.22.37.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.2.keytab.
I20260620 18:38:54.200017 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.37.2:0
--local_ip_for_outbound_sockets=127.22.37.2
--webserver_interface=127.22.37.2
--webserver_port=0
--tserver_master_addrs=127.22.37.62:37219
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.2.keytab
--principal=kudu/127.22.37.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:36803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:38:54.520283 4271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:54.520599 4271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:54.520766 4271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:54.528434 4271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:54.528604 4271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:54.528795 4271 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.37.2
I20260620 18:38:54.538277 4271 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:36803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.2.keytab
--principal=kudu/127.22.37.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.37.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.37.2
--webserver_port=0
--tserver_master_addrs=127.22.37.62:37219
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.37.2
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:38:54.539543 4271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:54.541000 4271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:54.553805 4277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:54.554039 4279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:54.554330 4276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:54.555868 4271 server_base.cc:1034] running on GCE node
I20260620 18:38:54.556450 4271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:54.557327 4271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:54.558560 4271 hybrid_clock.cc:648] HybridClock initialized: now 1781980734558516 us; error 42 us; skew 500 ppm
I20260620 18:38:54.558841 4271 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:38:54 dist-test-slave-5lgp krb5kdc[4031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980734, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:54.561847 4271 init.cc:377] Logged in from keytab as kudu/127.22.37.2@KRBTEST.COM (short username kudu)
I20260620 18:38:54.563179 4271 webserver.cc:460] Webserver started at http://127.22.37.2:42065/ using document root <none> and password file <none>
I20260620 18:38:54.563908 4271 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:54.564047 4271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:54.564343 4271 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:54.567052 4271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-1/data/instance:
uuid: "a50c31a3297f4a65885627657f513f8b"
format_stamp: "Formatted at 2026-06-20 18:38:54 on dist-test-slave-5lgp"
server_key: "10aff9d94f942e7f09b9b7bedb183355"
server_key_iv: "27fe09a0db58516d2f3ed83f473d44b9"
server_key_version: "encryptionkey@0"
I20260620 18:38:54.567812 4271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-1/wal/instance:
uuid: "a50c31a3297f4a65885627657f513f8b"
format_stamp: "Formatted at 2026-06-20 18:38:54 on dist-test-slave-5lgp"
server_key: "10aff9d94f942e7f09b9b7bedb183355"
server_key_iv: "27fe09a0db58516d2f3ed83f473d44b9"
server_key_version: "encryptionkey@0"
I20260620 18:38:54.572583 4271 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 18:38:54.576074 4286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:54.577409 4271 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:38:54.577591 4271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-1/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-1/wal
uuid: "a50c31a3297f4a65885627657f513f8b"
format_stamp: "Formatted at 2026-06-20 18:38:54 on dist-test-slave-5lgp"
server_key: "10aff9d94f942e7f09b9b7bedb183355"
server_key_iv: "27fe09a0db58516d2f3ed83f473d44b9"
server_key_version: "encryptionkey@0"
I20260620 18:38:54.577795 4271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:54.590170 4271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:54.594272 4271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:54.594620 4271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:54.595608 4271 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:38:54.596941 4271 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:38:54.597056 4271 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:54.597208 4271 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:38:54.597280 4271 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:54.630136 4271 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.2:41639
I20260620 18:38:54.630182 4398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.2:41639 every 8 connection(s)
I20260620 18:38:54.631439 4271 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-1/data/info.pb
I20260620 18:38:54.639814 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 4271
I20260620 18:38:54.639999 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-1/wal/instance
I20260620 18:38:54.640466 22676 external_mini_cluster.cc:1428] Setting key 3a85d3f365be045523939d94f132197f
Jun 20 18:38:54 dist-test-slave-5lgp krb5kdc[4031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980734, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.2@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM
I20260620 18:38:54.652418 4267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:54.633991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.2:53225 (local address 127.22.37.62:37219)
0620 18:38:54.634187 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:54.634191 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:54.635218 (+ 1027us) server_negotiation.cc:407] Connection header received
0620 18:38:54.636963 (+ 1745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:54.636968 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:54.637037 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:54.637124 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:54.638727 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:54.639509 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:54.641268 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:54.641893 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:54.642028 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:54.645475 (+ 3447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:54.645485 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:54.645492 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:54.645539 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:54.649179 (+ 3640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:54.649980 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:54.649989 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:54.649995 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:54.650162 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:54.650709 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:54.650712 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:54.650714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:54.650981 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:54.651092 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:54.652007 (+ 915us) server_negotiation.cc:299] Negotiation successful
0620 18:38:54.652183 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 18:38:54.654054 4401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:54.634380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:37219 (local address 127.22.37.2:53225)
0620 18:38:54.635053 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:54.635097 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:54.636701 (+ 1604us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:54.637334 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:54.637342 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:54.637904 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:54.638490 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:54.638507 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:54.639652 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:54.639655 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:54.641113 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:54.641130 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:54.642097 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:54.642102 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:54.642391 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:54.643072 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:54.643107 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:54.645200 (+ 2093us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:54.649336 (+ 4136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:54.649343 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:54.649356 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:54.649855 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:54.650302 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:54.650306 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:54.650308 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:54.650589 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:54.651109 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:54.651117 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:54.651600 (+ 483us) client_negotiation.cc:769] Sending connection context
0620 18:38:54.651925 (+ 325us) client_negotiation.cc:240] Negotiation successful
0620 18:38:54.652529 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":254,"threads_started":1}
I20260620 18:38:54.655598 4399 heartbeater.cc:347] Connected to a master server at 127.22.37.62:37219
I20260620 18:38:54.655936 4399 heartbeater.cc:464] Registering TS with master...
I20260620 18:38:54.656663 4399 heartbeater.cc:511] Master 127.22.37.62:37219 requested a full tablet report, sending...
I20260620 18:38:54.658141 4076 ts_manager.cc:194] Registered new tserver with Master: a50c31a3297f4a65885627657f513f8b (127.22.37.2:41639)
I20260620 18:38:54.659180 4076 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.37.2@KRBTEST.COM'} at 127.22.37.2:53225
WARNING: no policy specified for kudu/127.22.37.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.37.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.37.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.3.keytab.
Entry for principal kudu/127.22.37.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.3.keytab.
I20260620 18:38:54.692905 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.37.3:0
--local_ip_for_outbound_sockets=127.22.37.3
--webserver_interface=127.22.37.3
--webserver_port=0
--tserver_master_addrs=127.22.37.62:37219
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.3.keytab
--principal=kudu/127.22.37.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:36803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:38:55.018859 4404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:55.019217 4404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:55.019392 4404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:55.026727 4404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:55.026893 4404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:55.027081 4404 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.37.3
I20260620 18:38:55.036801 4404 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:36803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.3.keytab
--principal=kudu/127.22.37.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.37.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.37.3
--webserver_port=0
--tserver_master_addrs=127.22.37.62:37219
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.37.3
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:38:55.038218 4404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:55.039635 4404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:55.053504 4409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:55.053504 4412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:55.053540 4410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:55.054752 4404 server_base.cc:1034] running on GCE node
I20260620 18:38:55.055481 4404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:55.056463 4404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:55.057688 4404 hybrid_clock.cc:648] HybridClock initialized: now 1781980735057630 us; error 52 us; skew 500 ppm
I20260620 18:38:55.058022 4404 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:38:55 dist-test-slave-5lgp krb5kdc[4031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980735, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:55.061319 4404 init.cc:377] Logged in from keytab as kudu/127.22.37.3@KRBTEST.COM (short username kudu)
I20260620 18:38:55.062836 4404 webserver.cc:460] Webserver started at http://127.22.37.3:43507/ using document root <none> and password file <none>
I20260620 18:38:55.063544 4404 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:55.063692 4404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:55.064023 4404 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:55.067379 4404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-2/data/instance:
uuid: "113f3aa889864bd291885fb8a53e7f79"
format_stamp: "Formatted at 2026-06-20 18:38:55 on dist-test-slave-5lgp"
server_key: "e9f76a09726ed0c1e0a9d5bdca8d083f"
server_key_iv: "53f9ff7ba04381dfa98bf4cbf99fa2c0"
server_key_version: "encryptionkey@0"
I20260620 18:38:55.068169 4404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-2/wal/instance:
uuid: "113f3aa889864bd291885fb8a53e7f79"
format_stamp: "Formatted at 2026-06-20 18:38:55 on dist-test-slave-5lgp"
server_key: "e9f76a09726ed0c1e0a9d5bdca8d083f"
server_key_iv: "53f9ff7ba04381dfa98bf4cbf99fa2c0"
server_key_version: "encryptionkey@0"
I20260620 18:38:55.073213 4404 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 18:38:55.076840 4419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:55.078315 4404 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:38:55.078476 4404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-2/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-2/wal
uuid: "113f3aa889864bd291885fb8a53e7f79"
format_stamp: "Formatted at 2026-06-20 18:38:55 on dist-test-slave-5lgp"
server_key: "e9f76a09726ed0c1e0a9d5bdca8d083f"
server_key_iv: "53f9ff7ba04381dfa98bf4cbf99fa2c0"
server_key_version: "encryptionkey@0"
I20260620 18:38:55.078685 4404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:55.093990 4404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:55.097826 4404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:55.098104 4404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:55.099016 4404 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:38:55.100309 4404 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:38:55.100378 4404 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:55.100531 4404 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:38:55.100585 4404 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:55.133525 4404 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.3:34609
I20260620 18:38:55.133529 4531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.3:34609 every 8 connection(s)
I20260620 18:38:55.134815 4404 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-2/data/info.pb
I20260620 18:38:55.135288 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 4404
I20260620 18:38:55.135476 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-2/wal/instance
I20260620 18:38:55.136116 22676 external_mini_cluster.cc:1428] Setting key c3dd40235844faebca83ff97e0a72215
Jun 20 18:38:55 dist-test-slave-5lgp krb5kdc[4031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980735, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.3@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM
I20260620 18:38:55.157014 4267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:55.138171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.3:45655 (local address 127.22.37.62:37219)
0620 18:38:55.138443 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:55.138473 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:55.139908 (+ 1435us) server_negotiation.cc:407] Connection header received
0620 18:38:55.142646 (+ 2738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:55.142651 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:55.142791 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:55.142905 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:55.144751 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:55.145900 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:55.147364 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:55.147768 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:55.147858 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:55.151065 (+ 3207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:55.151078 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:55.151082 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:55.151130 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:55.154394 (+ 3264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:55.155078 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:55.155086 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:55.155092 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:55.155256 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:55.155624 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:55.155627 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:55.155629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:55.155888 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:55.155997 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:55.156614 (+ 617us) server_negotiation.cc:299] Negotiation successful
0620 18:38:55.156782 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260620 18:38:55.158015 4534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:55.138814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:37219 (local address 127.22.37.3:45655)
0620 18:38:55.139749 (+ 935us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:55.139802 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:55.142260 (+ 2458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:55.143342 (+ 1082us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:55.143356 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:55.144029 (+ 673us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:55.144587 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:55.144604 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:55.146039 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:55.146043 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:55.147236 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:55.147248 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:55.147939 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:55.147943 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:55.148134 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:55.148717 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:55.148738 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:55.150868 (+ 2130us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:55.154514 (+ 3646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:55.154521 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:55.154532 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:55.154964 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:55.155351 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:55.155354 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:55.155356 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:55.155534 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:55.155989 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:55.155995 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:55.156288 (+ 293us) client_negotiation.cc:769] Sending connection context
0620 18:38:55.156524 (+ 236us) client_negotiation.cc:240] Negotiation successful
0620 18:38:55.156903 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":339,"threads_started":1}
I20260620 18:38:55.159324 4532 heartbeater.cc:347] Connected to a master server at 127.22.37.62:37219
I20260620 18:38:55.159613 4532 heartbeater.cc:464] Registering TS with master...
I20260620 18:38:55.160322 4532 heartbeater.cc:511] Master 127.22.37.62:37219 requested a full tablet report, sending...
I20260620 18:38:55.161713 4076 ts_manager.cc:194] Registered new tserver with Master: 113f3aa889864bd291885fb8a53e7f79 (127.22.37.3:34609)
I20260620 18:38:55.162514 4076 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.37.3@KRBTEST.COM'} at 127.22.37.3:45655
I20260620 18:38:55.165766 22676 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:38:55.174010 4265 heartbeater.cc:503] Master 127.22.37.62:37219 was elected leader, sending a full tablet report...
I20260620 18:38:55.182945 4267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:55.170161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51480 (local address 127.22.37.62:37219)
0620 18:38:55.170358 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:55.170366 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:55.170746 (+ 380us) server_negotiation.cc:407] Connection header received
0620 18:38:55.170933 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:55.170939 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:55.171017 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:55.171119 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:55.172467 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:55.173204 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:55.174653 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:55.175188 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:55.175319 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:55.177588 (+ 2269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:55.177594 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:55.177598 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:55.177636 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:55.180146 (+ 2510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:55.180805 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:55.180812 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:55.180815 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:55.180945 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:55.181354 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:55.181359 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:55.181362 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:55.181793 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:55.182055 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:55.182406 (+ 351us) server_negotiation.cc:299] Negotiation successful
0620 18:38:55.182623 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 18:38:55.186630 4076 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51480:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:38:55.193619 4076 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:38:55.226999 4545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:55.205583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60282 (local address 127.22.37.2:41639)
0620 18:38:55.217116 (+ 11533us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:55.217128 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:55.217162 (+ 34us) server_negotiation.cc:407] Connection header received
0620 18:38:55.217287 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:55.217298 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:55.217690 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:55.217939 (+ 249us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:38:55.218836 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:55.220966 (+ 2130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:55.224496 (+ 3530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:55.225837 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:55.225990 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:55.226201 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:55.226508 (+ 307us) server_negotiation.cc:299] Negotiation successful
0620 18:38:55.226671 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11361,"thread_start_us":11201,"threads_started":1}
I20260620 18:38:55.227044 4546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:55.205490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.2:41639 (local address 127.0.0.1:60282)
0620 18:38:55.215636 (+ 10146us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:55.215699 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:55.215964 (+ 265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:55.217985 (+ 2021us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:55.217997 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:55.218039 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:38:55.218590 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:55.218606 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:55.221194 (+ 2588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:55.221203 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:55.224310 (+ 3107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:55.224324 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:55.226075 (+ 1751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:55.226084 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:55.226278 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:55.226319 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 18:38:55.226456 (+ 137us) client_negotiation.cc:240] Negotiation successful
0620 18:38:55.226648 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9800,"thread_start_us":10643,"threads_started":1}
I20260620 18:38:55.230283 4548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:55.216865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34356 (local address 127.22.37.3:34609)
0620 18:38:55.221713 (+ 4848us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:55.221722 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:55.222568 (+ 846us) server_negotiation.cc:407] Connection header received
0620 18:38:55.222674 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:55.222681 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:55.222900 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:55.223091 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:38:55.223970 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:55.225360 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:55.227863 (+ 2503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:55.229188 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:55.229330 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:55.229595 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:55.229797 (+ 202us) server_negotiation.cc:299] Negotiation successful
0620 18:38:55.229944 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4671,"thread_start_us":4498,"threads_started":1}
I20260620 18:38:55.230268 4549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:55.218353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.3:34609 (local address 127.0.0.1:34356)
0620 18:38:55.222311 (+ 3958us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:55.222331 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:55.222445 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:55.223257 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:55.223262 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:55.223295 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:38:55.223806 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:55.223815 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:55.225525 (+ 1710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:55.225530 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:55.227709 (+ 2179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:55.227722 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:55.229412 (+ 1690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:55.229419 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:55.229585 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:55.229622 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 18:38:55.229735 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 18:38:55.229860 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3460,"mutex_wait_us":25,"thread_start_us":3372,"threads_started":1}
I20260620 18:38:55.231467 4334 tablet_service.cc:1467] Processing CreateTablet for tablet 373403a2b13a42a9983849a50850e96f (DEFAULT_TABLE table=test-table [id=f3c2bf7076e44df88e93797a68e95a87]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:38:55.232646 4334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 373403a2b13a42a9983849a50850e96f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:55.233114 4467 tablet_service.cc:1467] Processing CreateTablet for tablet 373403a2b13a42a9983849a50850e96f (DEFAULT_TABLE table=test-table [id=f3c2bf7076e44df88e93797a68e95a87]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:38:55.234299 4467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 373403a2b13a42a9983849a50850e96f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:55.235337 4544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:55.206124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.1:39491 (local address 127.0.0.1:59058)
0620 18:38:55.216402 (+ 10278us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:55.216423 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:55.216553 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:55.222982 (+ 6429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:55.222987 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:55.223010 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:38:55.223520 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:55.223532 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:55.225163 (+ 1631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:55.225170 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:55.227396 (+ 2226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:55.227409 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:55.234509 (+ 7100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:55.234516 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:55.234693 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:55.234726 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 18:38:55.234831 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 18:38:55.234967 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10143,"mutex_wait_us":25,"thread_start_us":8982,"threads_started":1}
I20260620 18:38:55.235389 4547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:55.206833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59058 (local address 127.22.37.1:39491)
0620 18:38:55.222343 (+ 15510us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:55.222351 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:55.222381 (+ 30us) server_negotiation.cc:407] Connection header received
0620 18:38:55.222489 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:55.222496 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:55.222774 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:55.222986 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:38:55.223766 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:55.224998 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:55.227561 (+ 2563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:55.228570 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:55.228723 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:55.228952 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:55.234896 (+ 5944us) server_negotiation.cc:299] Negotiation successful
0620 18:38:55.235073 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15319,"thread_start_us":15192,"threads_started":1}
I20260620 18:38:55.239100 4200 tablet_service.cc:1467] Processing CreateTablet for tablet 373403a2b13a42a9983849a50850e96f (DEFAULT_TABLE table=test-table [id=f3c2bf7076e44df88e93797a68e95a87]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:38:55.240444 4200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 373403a2b13a42a9983849a50850e96f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:55.245612 4550 tablet_bootstrap.cc:492] T 373403a2b13a42a9983849a50850e96f P 113f3aa889864bd291885fb8a53e7f79: Bootstrap starting.
I20260620 18:38:55.249445 4550 tablet_bootstrap.cc:654] T 373403a2b13a42a9983849a50850e96f P 113f3aa889864bd291885fb8a53e7f79: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:55.250118 4551 tablet_bootstrap.cc:492] T 373403a2b13a42a9983849a50850e96f P a50c31a3297f4a65885627657f513f8b: Bootstrap starting.
I20260620 18:38:55.250818 4550 log.cc:824] T 373403a2b13a42a9983849a50850e96f P 113f3aa889864bd291885fb8a53e7f79: Log is configured to *not* fsync() on all Append() calls
I20260620 18:38:55.253580 4551 tablet_bootstrap.cc:654] T 373403a2b13a42a9983849a50850e96f P a50c31a3297f4a65885627657f513f8b: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:55.253984 4550 tablet_bootstrap.cc:492] T 373403a2b13a42a9983849a50850e96f P 113f3aa889864bd291885fb8a53e7f79: No bootstrap required, opened a new log
I20260620 18:38:55.254251 4550 ts_tablet_manager.cc:1397] T 373403a2b13a42a9983849a50850e96f P 113f3aa889864bd291885fb8a53e7f79: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260620 18:38:55.254882 4551 log.cc:824] T 373403a2b13a42a9983849a50850e96f P a50c31a3297f4a65885627657f513f8b: Log is configured to *not* fsync() on all Append() calls
I20260620 18:38:55.255465 4553 tablet_bootstrap.cc:492] T 373403a2b13a42a9983849a50850e96f P aa1a40a28b54480c80bbf29fe96919a7: Bootstrap starting.
I20260620 18:38:55.258831 4553 tablet_bootstrap.cc:654] T 373403a2b13a42a9983849a50850e96f P aa1a40a28b54480c80bbf29fe96919a7: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:55.259840 4553 log.cc:824] T 373403a2b13a42a9983849a50850e96f P aa1a40a28b54480c80bbf29fe96919a7: Log is configured to *not* fsync() on all Append() calls
I20260620 18:38:55.260103 4551 tablet_bootstrap.cc:492] T 373403a2b13a42a9983849a50850e96f P a50c31a3297f4a65885627657f513f8b: No bootstrap required, opened a new log
I20260620 18:38:55.260428 4551 ts_tablet_manager.cc:1397] T 373403a2b13a42a9983849a50850e96f P a50c31a3297f4a65885627657f513f8b: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.009s
I20260620 18:38:55.262557 4553 tablet_bootstrap.cc:492] T 373403a2b13a42a9983849a50850e96f P aa1a40a28b54480c80bbf29fe96919a7: No bootstrap required, opened a new log
I20260620 18:38:55.262809 4553 ts_tablet_manager.cc:1397] T 373403a2b13a42a9983849a50850e96f P aa1a40a28b54480c80bbf29fe96919a7: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260620 18:38:55.265184 4550 raft_consensus.cc:348] T 373403a2b13a42a9983849a50850e96f P 113f3aa889864bd291885fb8a53e7f79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a50c31a3297f4a65885627657f513f8b" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 41639 } } peers { permanent_uuid: "113f3aa889864bd291885fb8a53e7f79" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 34609 } } peers { permanent_uuid: "aa1a40a28b54480c80bbf29fe96919a7" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 39491 } }
I20260620 18:38:55.265674 4550 raft_consensus.cc:374] T 373403a2b13a42a9983849a50850e96f P 113f3aa889864bd291885fb8a53e7f79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:55.265776 4550 raft_consensus.cc:729] T 373403a2b13a42a9983849a50850e96f P 113f3aa889864bd291885fb8a53e7f79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 113f3aa889864bd291885fb8a53e7f79, State: Initialized, Role: FOLLOWER
I20260620 18:38:55.266397 4550 consensus_queue.cc:260] T 373403a2b13a42a9983849a50850e96f P 113f3aa889864bd291885fb8a53e7f79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a50c31a3297f4a65885627657f513f8b" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 41639 } } peers { permanent_uuid: "113f3aa889864bd291885fb8a53e7f79" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 34609 } } peers { permanent_uuid: "aa1a40a28b54480c80bbf29fe96919a7" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 39491 } }
I20260620 18:38:55.267488 4532 heartbeater.cc:503] Master 127.22.37.62:37219 was elected leader, sending a full tablet report...
I20260620 18:38:55.268101 4550 ts_tablet_manager.cc:1428] T 373403a2b13a42a9983849a50850e96f P 113f3aa889864bd291885fb8a53e7f79: Time spent starting tablet: real 0.014s user 0.013s sys 0.001s
I20260620 18:38:55.271062 4551 raft_consensus.cc:348] T 373403a2b13a42a9983849a50850e96f P a50c31a3297f4a65885627657f513f8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a50c31a3297f4a65885627657f513f8b" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 41639 } } peers { permanent_uuid: "113f3aa889864bd291885fb8a53e7f79" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 34609 } } peers { permanent_uuid: "aa1a40a28b54480c80bbf29fe96919a7" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 39491 } }
I20260620 18:38:55.271476 4551 raft_consensus.cc:374] T 373403a2b13a42a9983849a50850e96f P a50c31a3297f4a65885627657f513f8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:55.271571 4551 raft_consensus.cc:729] T 373403a2b13a42a9983849a50850e96f P a50c31a3297f4a65885627657f513f8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a50c31a3297f4a65885627657f513f8b, State: Initialized, Role: FOLLOWER
I20260620 18:38:55.272137 4551 consensus_queue.cc:260] T 373403a2b13a42a9983849a50850e96f P a50c31a3297f4a65885627657f513f8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a50c31a3297f4a65885627657f513f8b" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 41639 } } peers { permanent_uuid: "113f3aa889864bd291885fb8a53e7f79" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 34609 } } peers { permanent_uuid: "aa1a40a28b54480c80bbf29fe96919a7" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 39491 } }
I20260620 18:38:55.273125 4399 heartbeater.cc:503] Master 127.22.37.62:37219 was elected leader, sending a full tablet report...
I20260620 18:38:55.273044 4553 raft_consensus.cc:348] T 373403a2b13a42a9983849a50850e96f P aa1a40a28b54480c80bbf29fe96919a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a50c31a3297f4a65885627657f513f8b" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 41639 } } peers { permanent_uuid: "113f3aa889864bd291885fb8a53e7f79" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 34609 } } peers { permanent_uuid: "aa1a40a28b54480c80bbf29fe96919a7" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 39491 } }
I20260620 18:38:55.273483 4553 raft_consensus.cc:374] T 373403a2b13a42a9983849a50850e96f P aa1a40a28b54480c80bbf29fe96919a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:55.273643 4553 raft_consensus.cc:729] T 373403a2b13a42a9983849a50850e96f P aa1a40a28b54480c80bbf29fe96919a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa1a40a28b54480c80bbf29fe96919a7, State: Initialized, Role: FOLLOWER
I20260620 18:38:55.273914 4551 ts_tablet_manager.cc:1428] T 373403a2b13a42a9983849a50850e96f P a50c31a3297f4a65885627657f513f8b: Time spent starting tablet: real 0.013s user 0.012s sys 0.001s
I20260620 18:38:55.274263 4553 consensus_queue.cc:260] T 373403a2b13a42a9983849a50850e96f P aa1a40a28b54480c80bbf29fe96919a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a50c31a3297f4a65885627657f513f8b" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 41639 } } peers { permanent_uuid: "113f3aa889864bd291885fb8a53e7f79" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 34609 } } peers { permanent_uuid: "aa1a40a28b54480c80bbf29fe96919a7" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 39491 } }
I20260620 18:38:55.275924 4553 ts_tablet_manager.cc:1428] T 373403a2b13a42a9983849a50850e96f P aa1a40a28b54480c80bbf29fe96919a7: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
I20260620 18:38:55.345808 4557 raft_consensus.cc:482] T 373403a2b13a42a9983849a50850e96f P a50c31a3297f4a65885627657f513f8b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:38:55.346133 4557 raft_consensus.cc:504] T 373403a2b13a42a9983849a50850e96f P a50c31a3297f4a65885627657f513f8b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a50c31a3297f4a65885627657f513f8b" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 41639 } } peers { permanent_uuid: "113f3aa889864bd291885fb8a53e7f79" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 34609 } } peers { permanent_uuid: "aa1a40a28b54480c80bbf29fe96919a7" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 39491 } }
I20260620 18:38:55.347764 4557 leader_election.cc:288] T 373403a2b13a42a9983849a50850e96f P a50c31a3297f4a65885627657f513f8b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 113f3aa889864bd291885fb8a53e7f79 (127.22.37.3:34609), aa1a40a28b54480c80bbf29fe96919a7 (127.22.37.1:39491)
I20260620 18:38:55.354712 4548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:55.348266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.2:39417 (local address 127.22.37.3:34609)
0620 18:38:55.348478 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:55.348485 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:55.349264 (+ 779us) server_negotiation.cc:407] Connection header received
0620 18:38:55.349470 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:55.349476 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:55.349563 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:55.349667 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:38:55.350322 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:55.351076 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:55.352874 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:55.353717 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:55.353814 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:55.353943 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:55.354296 (+ 353us) server_negotiation.cc:299] Negotiation successful
0620 18:38:55.354451 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 18:38:55.354743 4547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:55.348319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.2:42313 (local address 127.22.37.1:39491)
0620 18:38:55.348532 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:55.348540 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:55.349258 (+ 718us) server_negotiation.cc:407] Connection header received
0620 18:38:55.349580 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:55.349584 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:55.349675 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:55.349763 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:38:55.350421 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:55.351121 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:55.352914 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:55.353750 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:55.353922 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:55.354022 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:55.354380 (+ 358us) server_negotiation.cc:299] Negotiation successful
0620 18:38:55.354522 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 18:38:55.354797 4559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:55.348165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.3:34609 (local address 127.22.37.2:39417)
0620 18:38:55.349134 (+ 969us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:55.349148 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:55.349292 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:55.349699 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:55.349707 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:55.349740 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:38:55.350117 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:55.350143 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:55.351205 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:55.351213 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:55.352718 (+ 1505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:55.352729 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:55.353941 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:55.353947 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:55.354138 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:55.354165 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 18:38:55.354237 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 18:38:55.354364 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":864,"mutex_wait_us":1,"thread_start_us":517,"threads_started":1}
I20260620 18:38:55.354902 4560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:55.348075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.1:39491 (local address 127.22.37.2:42313)
0620 18:38:55.349111 (+ 1036us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:55.349135 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:55.349327 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:55.349802 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:55.349809 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:55.349857 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:38:55.350273 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:55.350283 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:55.351275 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:55.351278 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:55.352729 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:55.352741 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:55.354006 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:55.354011 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:55.354193 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:55.354237 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 18:38:55.354331 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 18:38:55.354450 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":862,"mutex_wait_us":79,"spinlock_wait_cycles":23808,"thread_start_us":622,"threads_started":1}
I20260620 18:38:55.355978 4487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "373403a2b13a42a9983849a50850e96f" candidate_uuid: "a50c31a3297f4a65885627657f513f8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "113f3aa889864bd291885fb8a53e7f79" is_pre_election: true
I20260620 18:38:55.355980 4220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "373403a2b13a42a9983849a50850e96f" candidate_uuid: "a50c31a3297f4a65885627657f513f8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa1a40a28b54480c80bbf29fe96919a7" is_pre_election: true
I20260620 18:38:55.356566 4220 raft_consensus.cc:2454] T 373403a2b13a42a9983849a50850e96f P aa1a40a28b54480c80bbf29fe96919a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a50c31a3297f4a65885627657f513f8b in term 0.
I20260620 18:38:55.356566 4487 raft_consensus.cc:2454] T 373403a2b13a42a9983849a50850e96f P 113f3aa889864bd291885fb8a53e7f79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a50c31a3297f4a65885627657f513f8b in term 0.
I20260620 18:38:55.357290 4288 leader_election.cc:302] T 373403a2b13a42a9983849a50850e96f P a50c31a3297f4a65885627657f513f8b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 113f3aa889864bd291885fb8a53e7f79, a50c31a3297f4a65885627657f513f8b, aa1a40a28b54480c80bbf29fe96919a7; no voters:
I20260620 18:38:55.357842 4557 raft_consensus.cc:2781] T 373403a2b13a42a9983849a50850e96f P a50c31a3297f4a65885627657f513f8b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:38:55.358022 4557 raft_consensus.cc:482] T 373403a2b13a42a9983849a50850e96f P a50c31a3297f4a65885627657f513f8b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:38:55.358148 4557 raft_consensus.cc:3037] T 373403a2b13a42a9983849a50850e96f P a50c31a3297f4a65885627657f513f8b [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:55.360553 4557 raft_consensus.cc:504] T 373403a2b13a42a9983849a50850e96f P a50c31a3297f4a65885627657f513f8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a50c31a3297f4a65885627657f513f8b" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 41639 } } peers { permanent_uuid: "113f3aa889864bd291885fb8a53e7f79" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 34609 } } peers { permanent_uuid: "aa1a40a28b54480c80bbf29fe96919a7" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 39491 } }
I20260620 18:38:55.361127 4557 leader_election.cc:288] T 373403a2b13a42a9983849a50850e96f P a50c31a3297f4a65885627657f513f8b [CANDIDATE]: Term 1 election: Requested vote from peers 113f3aa889864bd291885fb8a53e7f79 (127.22.37.3:34609), aa1a40a28b54480c80bbf29fe96919a7 (127.22.37.1:39491)
I20260620 18:38:55.361626 4220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "373403a2b13a42a9983849a50850e96f" candidate_uuid: "a50c31a3297f4a65885627657f513f8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa1a40a28b54480c80bbf29fe96919a7"
I20260620 18:38:55.361626 4487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "373403a2b13a42a9983849a50850e96f" candidate_uuid: "a50c31a3297f4a65885627657f513f8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "113f3aa889864bd291885fb8a53e7f79"
I20260620 18:38:55.361836 4220 raft_consensus.cc:3037] T 373403a2b13a42a9983849a50850e96f P aa1a40a28b54480c80bbf29fe96919a7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:55.361836 4487 raft_consensus.cc:3037] T 373403a2b13a42a9983849a50850e96f P 113f3aa889864bd291885fb8a53e7f79 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:55.365037 4487 raft_consensus.cc:2454] T 373403a2b13a42a9983849a50850e96f P 113f3aa889864bd291885fb8a53e7f79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a50c31a3297f4a65885627657f513f8b in term 1.
I20260620 18:38:55.365039 4220 raft_consensus.cc:2454] T 373403a2b13a42a9983849a50850e96f P aa1a40a28b54480c80bbf29fe96919a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a50c31a3297f4a65885627657f513f8b in term 1.
I20260620 18:38:55.365461 4288 leader_election.cc:302] T 373403a2b13a42a9983849a50850e96f P a50c31a3297f4a65885627657f513f8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 113f3aa889864bd291885fb8a53e7f79, a50c31a3297f4a65885627657f513f8b; no voters:
I20260620 18:38:55.365756 4557 raft_consensus.cc:2781] T 373403a2b13a42a9983849a50850e96f P a50c31a3297f4a65885627657f513f8b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:38:55.366226 4557 raft_consensus.cc:686] T 373403a2b13a42a9983849a50850e96f P a50c31a3297f4a65885627657f513f8b [term 1 LEADER]: Becoming Leader. State: Replica: a50c31a3297f4a65885627657f513f8b, State: Running, Role: LEADER
I20260620 18:38:55.366671 4557 consensus_queue.cc:237] T 373403a2b13a42a9983849a50850e96f P a50c31a3297f4a65885627657f513f8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a50c31a3297f4a65885627657f513f8b" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 41639 } } peers { permanent_uuid: "113f3aa889864bd291885fb8a53e7f79" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 34609 } } peers { permanent_uuid: "aa1a40a28b54480c80bbf29fe96919a7" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 39491 } }
I20260620 18:38:55.371403 4076 catalog_manager.cc:5495] T 373403a2b13a42a9983849a50850e96f P a50c31a3297f4a65885627657f513f8b reported cstate change: term changed from 0 to 1, leader changed from <none> to a50c31a3297f4a65885627657f513f8b (127.22.37.2). New cstate: current_term: 1 leader_uuid: "a50c31a3297f4a65885627657f513f8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a50c31a3297f4a65885627657f513f8b" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 41639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "113f3aa889864bd291885fb8a53e7f79" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 34609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aa1a40a28b54480c80bbf29fe96919a7" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 39491 } health_report { overall_health: UNKNOWN } } }
W20260620 18:38:55.383374 4400 tablet.cc:2369] T 373403a2b13a42a9983849a50850e96f P a50c31a3297f4a65885627657f513f8b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:38:55.386500 4533 tablet.cc:2369] T 373403a2b13a42a9983849a50850e96f P 113f3aa889864bd291885fb8a53e7f79: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:38:55.393991 4266 tablet.cc:2369] T 373403a2b13a42a9983849a50850e96f P aa1a40a28b54480c80bbf29fe96919a7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:38:55.407681 4545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:55.402179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60296 (local address 127.22.37.2:41639)
0620 18:38:55.402372 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:55.402379 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:55.402476 (+ 97us) server_negotiation.cc:407] Connection header received
0620 18:38:55.402715 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:55.402718 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:55.402796 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:55.402957 (+ 161us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:38:55.403355 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:55.404093 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:55.405556 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:55.405967 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:55.406105 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:55.406606 (+ 501us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:38:55.407080 (+ 474us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:38:55.407247 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:55.407298 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 18:38:55.407378 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 18:38:55.415611 4487 raft_consensus.cc:1261] T 373403a2b13a42a9983849a50850e96f P 113f3aa889864bd291885fb8a53e7f79 [term 1 FOLLOWER]: Refusing update from remote peer a50c31a3297f4a65885627657f513f8b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:38:55.415676 4220 raft_consensus.cc:1261] T 373403a2b13a42a9983849a50850e96f P aa1a40a28b54480c80bbf29fe96919a7 [term 1 FOLLOWER]: Refusing update from remote peer a50c31a3297f4a65885627657f513f8b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:38:55.416558 4561 consensus_queue.cc:1035] T 373403a2b13a42a9983849a50850e96f P a50c31a3297f4a65885627657f513f8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa1a40a28b54480c80bbf29fe96919a7" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 39491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:38:55.416901 4557 consensus_queue.cc:1035] T 373403a2b13a42a9983849a50850e96f P a50c31a3297f4a65885627657f513f8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "113f3aa889864bd291885fb8a53e7f79" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 34609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:38:55.429214 4566 mvcc.cc:205] Tried to move back new op lower bound from 7298993092252598272 to 7298993092065902592. Current Snapshot: MvccSnapshot[applied={T|T < 7298993092252598272}]
I20260620 18:38:55.429510 4565 mvcc.cc:205] Tried to move back new op lower bound from 7298993092252598272 to 7298993092065902592. Current Snapshot: MvccSnapshot[applied={T|T < 7298993092252598272}]
I20260620 18:38:55.441632 4076 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51480:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:38:55.441912 4076 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51480:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:38:55.445653 4076 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b7f7b758d94d44a19e5997546da74de4: Sending DeleteTablet for 3 replicas of tablet 373403a2b13a42a9983849a50850e96f
I20260620 18:38:55.446902 4334 tablet_service.cc:1514] Processing DeleteTablet for tablet 373403a2b13a42a9983849a50850e96f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:38:55 UTC) from {username='kudu', principal='kudu/127.22.37.62@KRBTEST.COM'} at 127.0.0.1:60282
I20260620 18:38:55.446990 4467 tablet_service.cc:1514] Processing DeleteTablet for tablet 373403a2b13a42a9983849a50850e96f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:38:55 UTC) from {username='kudu', principal='kudu/127.22.37.62@KRBTEST.COM'} at 127.0.0.1:34356
I20260620 18:38:55.447544 4200 tablet_service.cc:1514] Processing DeleteTablet for tablet 373403a2b13a42a9983849a50850e96f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:38:55 UTC) from {username='kudu', principal='kudu/127.22.37.62@KRBTEST.COM'} at 127.0.0.1:59058
I20260620 18:38:55.448417 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 4137
I20260620 18:38:55.457847 4576 tablet_replica.cc:330] T 373403a2b13a42a9983849a50850e96f P aa1a40a28b54480c80bbf29fe96919a7: stopping tablet replica
I20260620 18:38:55.458392 4576 raft_consensus.cc:2229] T 373403a2b13a42a9983849a50850e96f P aa1a40a28b54480c80bbf29fe96919a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:38:55.459081 4576 raft_consensus.cc:2258] T 373403a2b13a42a9983849a50850e96f P aa1a40a28b54480c80bbf29fe96919a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:38:55.459411 4574 tablet_replica.cc:330] T 373403a2b13a42a9983849a50850e96f P a50c31a3297f4a65885627657f513f8b: stopping tablet replica
I20260620 18:38:55.462316 4576 ts_tablet_manager.cc:1905] T 373403a2b13a42a9983849a50850e96f P aa1a40a28b54480c80bbf29fe96919a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:38:55.466917 4574 raft_consensus.cc:2229] T 373403a2b13a42a9983849a50850e96f P a50c31a3297f4a65885627657f513f8b [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:38:55.468387 4574 raft_consensus.cc:2258] T 373403a2b13a42a9983849a50850e96f P a50c31a3297f4a65885627657f513f8b [term 1 FOLLOWER]: Raft consensus is shut down!
Jun 20 18:38:55 dist-test-slave-5lgp krb5kdc[4031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980733, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.37.1@KRBTEST.COM
I20260620 18:38:55.472113 4576 ts_tablet_manager.cc:1918] T 373403a2b13a42a9983849a50850e96f P aa1a40a28b54480c80bbf29fe96919a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:38:55.472286 4574 ts_tablet_manager.cc:1905] T 373403a2b13a42a9983849a50850e96f P a50c31a3297f4a65885627657f513f8b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:38:55.472751 4575 tablet_replica.cc:330] T 373403a2b13a42a9983849a50850e96f P 113f3aa889864bd291885fb8a53e7f79: stopping tablet replica
I20260620 18:38:55.473249 4575 raft_consensus.cc:2229] T 373403a2b13a42a9983849a50850e96f P 113f3aa889864bd291885fb8a53e7f79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:38:55.473829 4575 raft_consensus.cc:2258] T 373403a2b13a42a9983849a50850e96f P 113f3aa889864bd291885fb8a53e7f79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:38:55.474275 4576 log.cc:1196] T 373403a2b13a42a9983849a50850e96f P aa1a40a28b54480c80bbf29fe96919a7: Deleting WAL directory at /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-0/wal/wals/373403a2b13a42a9983849a50850e96f
I20260620 18:38:55.474852 4576 ts_tablet_manager.cc:1939] T 373403a2b13a42a9983849a50850e96f P aa1a40a28b54480c80bbf29fe96919a7: Deleting consensus metadata
I20260620 18:38:55.476665 4575 ts_tablet_manager.cc:1905] T 373403a2b13a42a9983849a50850e96f P 113f3aa889864bd291885fb8a53e7f79: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:38:55.480222 4574 ts_tablet_manager.cc:1918] T 373403a2b13a42a9983849a50850e96f P a50c31a3297f4a65885627657f513f8b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:38:55.480450 4574 log.cc:1196] T 373403a2b13a42a9983849a50850e96f P a50c31a3297f4a65885627657f513f8b: Deleting WAL directory at /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-1/wal/wals/373403a2b13a42a9983849a50850e96f
I20260620 18:38:55.481158 4574 ts_tablet_manager.cc:1939] T 373403a2b13a42a9983849a50850e96f P a50c31a3297f4a65885627657f513f8b: Deleting consensus metadata
I20260620 18:38:55.481426 4547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:55.449952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59074 (local address 127.22.37.1:39491)
0620 18:38:55.450125 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:55.450131 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:55.457002 (+ 6871us) server_negotiation.cc:407] Connection header received
0620 18:38:55.457966 (+ 964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:55.457974 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:55.458068 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:55.458172 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:55.460675 (+ 2503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:55.461835 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:55.463952 (+ 2117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:55.465314 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:55.465478 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:55.473675 (+ 8197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:55.473690 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:55.473707 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:55.473745 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:55.477784 (+ 4039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:55.478741 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:55.478749 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:55.478753 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:55.478938 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:55.479408 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:55.479413 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:55.479417 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:55.479837 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:55.480125 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:55.480727 (+ 602us) server_negotiation.cc:299] Negotiation successful
0620 18:38:55.480948 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 18:38:55.482877 4260 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:38:55.483566 4061 catalog_manager.cc:4841] TS a50c31a3297f4a65885627657f513f8b (127.22.37.2:41639): tablet 373403a2b13a42a9983849a50850e96f (table test-table [id=f3c2bf7076e44df88e93797a68e95a87]) successfully deleted
I20260620 18:38:55.484429 4575 ts_tablet_manager.cc:1918] T 373403a2b13a42a9983849a50850e96f P 113f3aa889864bd291885fb8a53e7f79: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:38:55.484607 4575 log.cc:1196] T 373403a2b13a42a9983849a50850e96f P 113f3aa889864bd291885fb8a53e7f79: Deleting WAL directory at /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-2/wal/wals/373403a2b13a42a9983849a50850e96f
I20260620 18:38:55.485184 4575 ts_tablet_manager.cc:1939] T 373403a2b13a42a9983849a50850e96f P 113f3aa889864bd291885fb8a53e7f79: Deleting consensus metadata
I20260620 18:38:55.487003 4061 catalog_manager.cc:4841] TS 113f3aa889864bd291885fb8a53e7f79 (127.22.37.3:34609): tablet 373403a2b13a42a9983849a50850e96f (table test-table [id=f3c2bf7076e44df88e93797a68e95a87]) successfully deleted
I20260620 18:38:55.582600 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 4137
W20260620 18:38:55.602716 4060 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.37.1:39491 (error 108)
I20260620 18:38:55.602872 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 4271
I20260620 18:38:55.605494 4544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:55.604979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.1:39491 (local address 127.0.0.1:59088)
0620 18:38:55.605166 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:55.605293 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.1:39491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260620 18:38:55.605789 4060 catalog_manager.cc:4584] TS aa1a40a28b54480c80bbf29fe96919a7 (127.22.37.1:39491): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 373403a2b13a42a9983849a50850e96f: Network error: Client connection negotiation failed: client connection to 127.22.37.1:39491: connect: Connection refused (error 111)
Jun 20 18:38:55 dist-test-slave-5lgp krb5kdc[4031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980733, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.37.2@KRBTEST.COM
I20260620 18:38:55.621616 4545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:55.603782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60302 (local address 127.22.37.2:41639)
0620 18:38:55.603987 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:55.603993 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:55.604481 (+ 488us) server_negotiation.cc:407] Connection header received
0620 18:38:55.604572 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:55.604576 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:55.604652 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:55.604805 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:55.606707 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:55.608015 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:55.609639 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:55.610121 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:55.610232 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:55.615374 (+ 5142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:55.615389 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:55.615409 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:55.615451 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:55.618701 (+ 3250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:55.619433 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:55.619440 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:55.619443 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:55.619570 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:55.619971 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:55.619975 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:55.619977 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:55.620334 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:55.620697 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:55.621009 (+ 312us) server_negotiation.cc:299] Negotiation successful
0620 18:38:55.621238 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 18:38:55.622352 4394 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:38:55.659408 4544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:55.658936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.1:39491 (local address 127.0.0.1:59092)
0620 18:38:55.659155 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:55.659244 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.1:39491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260620 18:38:55.660537 4544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:55.660176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.1:39491 (local address 127.0.0.1:59100)
0620 18:38:55.660339 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:55.660419 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.1:39491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 18:38:55.721180 4544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:55.720681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.1:39491 (local address 127.0.0.1:59102)
0620 18:38:55.720904 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:55.721011 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.1:39491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260620 18:38:55.722246 4544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:55.721907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.1:39491 (local address 127.0.0.1:59118)
0620 18:38:55.722070 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:55.722132 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.1:39491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 18:38:55.735917 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 4271
I20260620 18:38:55.754469 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 4404
Jun 20 18:38:55 dist-test-slave-5lgp krb5kdc[4031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980733, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.37.3@KRBTEST.COM
I20260620 18:38:55.770128 4548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:55.755256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34368 (local address 127.22.37.3:34609)
0620 18:38:55.755438 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:55.755445 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:55.755519 (+ 74us) server_negotiation.cc:407] Connection header received
0620 18:38:55.755741 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:55.755745 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:55.755808 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:55.755911 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:55.757157 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:55.757996 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:55.759349 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:55.759748 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:55.759882 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:55.764337 (+ 4455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:55.764353 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:55.764369 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:55.764410 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:55.767296 (+ 2886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:55.767980 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:55.767988 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:55.767990 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:55.768132 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:55.768538 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:55.768545 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:55.768548 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:55.768914 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:55.769213 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:55.769564 (+ 351us) server_negotiation.cc:299] Negotiation successful
0620 18:38:55.769786 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 18:38:55.770769 4527 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:38:55.802901 4544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:55.802463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.1:39491 (local address 127.0.0.1:59134)
0620 18:38:55.802659 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:55.802746 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.1:39491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260620 18:38:55.803916 4544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:55.803532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.1:39491 (local address 127.0.0.1:59136)
0620 18:38:55.803725 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:55.803812 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.1:39491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260620 18:38:55.859637 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 4404
I20260620 18:38:55.876806 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 4044
I20260620 18:38:55.878034 4106 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:38:55.991025 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 4044
I20260620 18:38:56.012576 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.62:37219
--webserver_interface=127.22.37.62
--webserver_port=36715
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab
--principal=kudu/127.22.37.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:36803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.37.62:37219
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:38:56.339679 4583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:56.340031 4583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:56.340176 4583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:56.347347 4583 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:38:56.347450 4583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:56.347569 4583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:56.347694 4583 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:38:56.347776 4583 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:38:56.357549 4583 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:36803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.22.37.62:37219
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab
--principal=kudu/127.22.37.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.62:37219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.37.62
--webserver_port=36715
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:38:56.358843 4583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:56.360152 4583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:56.382344 4590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:56.382388 4589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:56.382364 4592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:56.383170 4583 server_base.cc:1034] running on GCE node
I20260620 18:38:56.384061 4583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:56.385663 4583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:56.386900 4583 hybrid_clock.cc:648] HybridClock initialized: now 1781980736386856 us; error 60 us; skew 500 ppm
I20260620 18:38:56.387305 4583 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:38:56 dist-test-slave-5lgp krb5kdc[4031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980736, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:56.390767 4583 init.cc:377] Logged in from keytab as kudu/127.22.37.62@KRBTEST.COM (short username kudu)
I20260620 18:38:56.392381 4583 webserver.cc:460] Webserver started at http://127.22.37.62:36715/ using document root <none> and password file <none>
I20260620 18:38:56.393141 4583 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:56.393303 4583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:56.399286 4583 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 18:38:56.402338 4599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:56.403680 4583 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:38:56.403877 4583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/master-0/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/master-0/wal
uuid: "b7f7b758d94d44a19e5997546da74de4"
format_stamp: "Formatted at 2026-06-20 18:38:53 on dist-test-slave-5lgp"
server_key: "b1b4626643bf9bc61dadb969b47e777c"
server_key_iv: "d22eee2a99893023ebdce3d2b992ae36"
server_key_version: "encryptionkey@0"
I20260620 18:38:56.404969 4583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:56.422750 4583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:56.426892 4583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:56.427196 4583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:56.446465 4583 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.62:37219
I20260620 18:38:56.446493 4650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.62:37219 every 8 connection(s)
I20260620 18:38:56.448027 4583 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
I20260620 18:38:56.455009 4651 sys_catalog.cc:263] Verifying existing consensus state
I20260620 18:38:56.455713 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 4583
I20260620 18:38:56.456696 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.37.1:39491
--local_ip_for_outbound_sockets=127.22.37.1
--tserver_master_addrs=127.22.37.62:37219
--webserver_port=41659
--webserver_interface=127.22.37.1
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.1.keytab
--principal=kudu/127.22.37.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:36803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:38:56.460100 4651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7f7b758d94d44a19e5997546da74de4: Bootstrap starting.
I20260620 18:38:56.482750 4651 log.cc:824] T 00000000000000000000000000000000 P b7f7b758d94d44a19e5997546da74de4: Log is configured to *not* fsync() on all Append() calls
I20260620 18:38:56.494549 4651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7f7b758d94d44a19e5997546da74de4: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 18:38:56.495057 4651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7f7b758d94d44a19e5997546da74de4: Bootstrap complete.
I20260620 18:38:56.504225 4651 raft_consensus.cc:348] T 00000000000000000000000000000000 P b7f7b758d94d44a19e5997546da74de4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7f7b758d94d44a19e5997546da74de4" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 37219 } }
I20260620 18:38:56.505051 4651 raft_consensus.cc:729] T 00000000000000000000000000000000 P b7f7b758d94d44a19e5997546da74de4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7f7b758d94d44a19e5997546da74de4, State: Initialized, Role: FOLLOWER
I20260620 18:38:56.505805 4651 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7f7b758d94d44a19e5997546da74de4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7f7b758d94d44a19e5997546da74de4" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 37219 } }
I20260620 18:38:56.506038 4651 raft_consensus.cc:388] T 00000000000000000000000000000000 P b7f7b758d94d44a19e5997546da74de4 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:38:56.506156 4651 raft_consensus.cc:482] T 00000000000000000000000000000000 P b7f7b758d94d44a19e5997546da74de4 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:38:56.506336 4651 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b7f7b758d94d44a19e5997546da74de4 [term 1 FOLLOWER]: Advancing to term 2
I20260620 18:38:56.510475 4651 raft_consensus.cc:504] T 00000000000000000000000000000000 P b7f7b758d94d44a19e5997546da74de4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7f7b758d94d44a19e5997546da74de4" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 37219 } }
I20260620 18:38:56.511020 4651 leader_election.cc:302] T 00000000000000000000000000000000 P b7f7b758d94d44a19e5997546da74de4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b7f7b758d94d44a19e5997546da74de4; no voters:
I20260620 18:38:56.511663 4651 leader_election.cc:288] T 00000000000000000000000000000000 P b7f7b758d94d44a19e5997546da74de4 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 18:38:56.511802 4655 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b7f7b758d94d44a19e5997546da74de4 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 18:38:56.513736 4655 raft_consensus.cc:686] T 00000000000000000000000000000000 P b7f7b758d94d44a19e5997546da74de4 [term 2 LEADER]: Becoming Leader. State: Replica: b7f7b758d94d44a19e5997546da74de4, State: Running, Role: LEADER
I20260620 18:38:56.514211 4651 sys_catalog.cc:564] T 00000000000000000000000000000000 P b7f7b758d94d44a19e5997546da74de4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:38:56.514334 4655 consensus_queue.cc:237] T 00000000000000000000000000000000 P b7f7b758d94d44a19e5997546da74de4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7f7b758d94d44a19e5997546da74de4" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 37219 } }
I20260620 18:38:56.518719 4656 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7f7b758d94d44a19e5997546da74de4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b7f7b758d94d44a19e5997546da74de4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7f7b758d94d44a19e5997546da74de4" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 37219 } } }
I20260620 18:38:56.519438 4656 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7f7b758d94d44a19e5997546da74de4 [sys.catalog]: This master's current role is: LEADER
I20260620 18:38:56.519084 4657 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7f7b758d94d44a19e5997546da74de4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b7f7b758d94d44a19e5997546da74de4. Latest consensus state: current_term: 2 leader_uuid: "b7f7b758d94d44a19e5997546da74de4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7f7b758d94d44a19e5997546da74de4" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 37219 } } }
I20260620 18:38:56.519893 4657 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7f7b758d94d44a19e5997546da74de4 [sys.catalog]: This master's current role is: LEADER
I20260620 18:38:56.521066 4664 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:38:56.535990 4664 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:38:56.537462 4664 catalog_manager.cc:1259] Loaded cluster ID: 001289474cef41599d8c617f2c4b8f7b
I20260620 18:38:56.537600 4664 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:38:56.540671 4664 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260620 18:38:56.558833 4664 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260620 18:38:56.581194 4664 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:38:56.582968 4664 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b7f7b758d94d44a19e5997546da74de4: Loaded TSK: 0
I20260620 18:38:56.584506 4664 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 18:38:56.820035 4653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:56.820354 4653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:56.820537 4653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:56.827654 4653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:56.827767 4653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:56.827942 4653 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.37.1
I20260620 18:38:56.837805 4653 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:36803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.1.keytab
--principal=kudu/127.22.37.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.37.1:39491
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.37.1
--webserver_port=41659
--tserver_master_addrs=127.22.37.62:37219
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.37.1
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:38:56.839190 4653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:56.840584 4653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:56.855976 4679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:56.856192 4678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:56.857069 4681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:56.857179 4653 server_base.cc:1034] running on GCE node
I20260620 18:38:56.858048 4653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:56.859390 4653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:56.860693 4653 hybrid_clock.cc:648] HybridClock initialized: now 1781980736860634 us; error 62 us; skew 500 ppm
I20260620 18:38:56.861101 4653 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:38:56 dist-test-slave-5lgp krb5kdc[4031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980736, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:56.868844 4653 init.cc:377] Logged in from keytab as kudu/127.22.37.1@KRBTEST.COM (short username kudu)
I20260620 18:38:56.870720 4653 webserver.cc:460] Webserver started at http://127.22.37.1:41659/ using document root <none> and password file <none>
I20260620 18:38:56.871559 4653 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:56.871701 4653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:56.878640 4653 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 18:38:56.881848 4688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:56.883458 4653 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 18:38:56.883630 4653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-0/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-0/wal
uuid: "aa1a40a28b54480c80bbf29fe96919a7"
format_stamp: "Formatted at 2026-06-20 18:38:54 on dist-test-slave-5lgp"
server_key: "c42730258bd5fb3f1e73310604833a5a"
server_key_iv: "43c786ff1cc383934b071c646e1298a1"
server_key_version: "encryptionkey@0"
I20260620 18:38:56.884913 4653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:56.905668 4653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:56.910275 4653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:56.910557 4653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:56.911595 4653 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:38:56.913033 4653 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:38:56.913163 4653 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:56.913298 4653 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:38:56.913443 4653 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:56.946974 4653 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.1:39491
I20260620 18:38:56.947058 4800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.1:39491 every 8 connection(s)
I20260620 18:38:56.948438 4653 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-0/data/info.pb
I20260620 18:38:56.953737 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 4653
I20260620 18:38:56.954562 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.37.2:41639
--local_ip_for_outbound_sockets=127.22.37.2
--tserver_master_addrs=127.22.37.62:37219
--webserver_port=42065
--webserver_interface=127.22.37.2
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.2.keytab
--principal=kudu/127.22.37.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:36803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 18:38:56 dist-test-slave-5lgp krb5kdc[4031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980736, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.1@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM
I20260620 18:38:56.975198 4803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:56.951238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.1:51005 (local address 127.22.37.62:37219)
0620 18:38:56.951893 (+ 655us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:56.951905 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:56.952361 (+ 456us) server_negotiation.cc:407] Connection header received
0620 18:38:56.955194 (+ 2833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:56.955208 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:56.955564 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:56.955860 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:56.957933 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:56.959154 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:56.961655 (+ 2501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:56.962792 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:56.962999 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:56.966288 (+ 3289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:56.966301 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:56.966321 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:56.966371 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:56.970516 (+ 4145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:56.971341 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:56.971352 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:56.971362 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:56.971589 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:56.972206 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:56.972212 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:56.972214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:56.972776 (+ 562us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:56.972957 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:56.973965 (+ 1008us) server_negotiation.cc:299] Negotiation successful
0620 18:38:56.974277 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":280,"threads_started":1}
I20260620 18:38:56.976006 4804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:56.951323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:37219 (local address 127.22.37.1:51005)
0620 18:38:56.952158 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:56.952203 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:56.954625 (+ 2422us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:56.956083 (+ 1458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:56.956097 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:56.956880 (+ 783us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:56.957694 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:56.957722 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:56.959329 (+ 1607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:56.959338 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:56.961515 (+ 2177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:56.961527 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:56.963000 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:56.963005 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:56.963202 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:56.963881 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:56.963902 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:56.965994 (+ 2092us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:56.970648 (+ 4654us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:56.970655 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:56.970666 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:56.971157 (+ 491us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:56.971736 (+ 579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:56.971744 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:56.971751 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:56.972015 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:56.972941 (+ 926us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:56.972953 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:56.973449 (+ 496us) client_negotiation.cc:769] Sending connection context
0620 18:38:56.973788 (+ 339us) client_negotiation.cc:240] Negotiation successful
0620 18:38:56.974417 (+ 629us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":358,"threads_started":1}
I20260620 18:38:56.979130 4801 heartbeater.cc:347] Connected to a master server at 127.22.37.62:37219
I20260620 18:38:56.979435 4801 heartbeater.cc:464] Registering TS with master...
I20260620 18:38:56.980149 4801 heartbeater.cc:511] Master 127.22.37.62:37219 requested a full tablet report, sending...
I20260620 18:38:56.982265 4616 ts_manager.cc:194] Registered new tserver with Master: aa1a40a28b54480c80bbf29fe96919a7 (127.22.37.1:39491)
I20260620 18:38:56.983794 4616 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.37.1@KRBTEST.COM'} at 127.22.37.1:51005
W20260620 18:38:57.286136 4805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:57.286577 4805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:57.286754 4805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:57.294054 4805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:57.294210 4805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:57.294400 4805 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.37.2
I20260620 18:38:57.304452 4805 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:36803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.2.keytab
--principal=kudu/127.22.37.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.37.2:41639
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.37.2
--webserver_port=42065
--tserver_master_addrs=127.22.37.62:37219
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.37.2
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:38:57.305858 4805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:57.307345 4805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:57.321856 4810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:57.321815 4813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:57.321931 4811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:57.323021 4805 server_base.cc:1034] running on GCE node
I20260620 18:38:57.323608 4805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:57.324566 4805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:57.325788 4805 hybrid_clock.cc:648] HybridClock initialized: now 1781980737325735 us; error 46 us; skew 500 ppm
I20260620 18:38:57.326108 4805 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:38:57 dist-test-slave-5lgp krb5kdc[4031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980737, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:57.329277 4805 init.cc:377] Logged in from keytab as kudu/127.22.37.2@KRBTEST.COM (short username kudu)
I20260620 18:38:57.330709 4805 webserver.cc:460] Webserver started at http://127.22.37.2:42065/ using document root <none> and password file <none>
I20260620 18:38:57.331367 4805 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:57.331519 4805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:57.336799 4805 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 18:38:57.339428 4820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:57.340646 4805 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 18:38:57.340813 4805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-1/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-1/wal
uuid: "a50c31a3297f4a65885627657f513f8b"
format_stamp: "Formatted at 2026-06-20 18:38:54 on dist-test-slave-5lgp"
server_key: "10aff9d94f942e7f09b9b7bedb183355"
server_key_iv: "27fe09a0db58516d2f3ed83f473d44b9"
server_key_version: "encryptionkey@0"
I20260620 18:38:57.342221 4805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:57.358655 4805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:57.362725 4805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:57.363015 4805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:57.364001 4805 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:38:57.365299 4805 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:38:57.365453 4805 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:57.365576 4805 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:38:57.365669 4805 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:57.399297 4805 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.2:41639
I20260620 18:38:57.399353 4932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.2:41639 every 8 connection(s)
I20260620 18:38:57.400776 4805 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-1/data/info.pb
I20260620 18:38:57.410733 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 4805
I20260620 18:38:57.411522 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.37.3:34609
--local_ip_for_outbound_sockets=127.22.37.3
--tserver_master_addrs=127.22.37.62:37219
--webserver_port=43507
--webserver_interface=127.22.37.3
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.3.keytab
--principal=kudu/127.22.37.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:36803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 18:38:57 dist-test-slave-5lgp krb5kdc[4031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980737, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.2@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM
I20260620 18:38:57.421305 4803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:57.403361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.2:53685 (local address 127.22.37.62:37219)
0620 18:38:57.403547 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:57.403551 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:57.404665 (+ 1114us) server_negotiation.cc:407] Connection header received
0620 18:38:57.406421 (+ 1756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:57.406425 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:57.406499 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:57.406586 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:57.408182 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:57.408931 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:57.410740 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:57.411410 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:57.411557 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:57.415256 (+ 3699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:57.415265 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:57.415270 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:57.415314 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:57.417990 (+ 2676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:57.418954 (+ 964us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:57.418959 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:57.418961 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:57.419093 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:57.419636 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:57.419640 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:57.419642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:57.419910 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:57.420074 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:57.420876 (+ 802us) server_negotiation.cc:299] Negotiation successful
0620 18:38:57.421059 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 18:38:57.422837 4935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:57.403825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:37219 (local address 127.22.37.2:53685)
0620 18:38:57.404514 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:57.404551 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:57.406168 (+ 1617us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:57.406821 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:57.406830 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:57.407386 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:57.407937 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:57.407953 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:57.409074 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:57.409078 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:57.410580 (+ 1502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:57.410605 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:57.411633 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:57.411643 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:57.411929 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:57.412607 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:57.412629 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:57.414969 (+ 2340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:57.418123 (+ 3154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:57.418131 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:57.418147 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:57.418770 (+ 623us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:57.419201 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:57.419208 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:57.419211 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:57.419468 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:57.420042 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:57.420052 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:57.420502 (+ 450us) client_negotiation.cc:769] Sending connection context
0620 18:38:57.420783 (+ 281us) client_negotiation.cc:240] Negotiation successful
0620 18:38:57.421391 (+ 608us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":274,"threads_started":1}
I20260620 18:38:57.424333 4933 heartbeater.cc:347] Connected to a master server at 127.22.37.62:37219
I20260620 18:38:57.424654 4933 heartbeater.cc:464] Registering TS with master...
I20260620 18:38:57.425421 4933 heartbeater.cc:511] Master 127.22.37.62:37219 requested a full tablet report, sending...
I20260620 18:38:57.426846 4616 ts_manager.cc:194] Registered new tserver with Master: a50c31a3297f4a65885627657f513f8b (127.22.37.2:41639)
I20260620 18:38:57.427655 4616 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.37.2@KRBTEST.COM'} at 127.22.37.2:53685
W20260620 18:38:57.739985 4936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:57.740300 4936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:57.740473 4936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:57.747702 4936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:57.747853 4936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:57.748025 4936 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.37.3
I20260620 18:38:57.757589 4936 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:36803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.3.keytab
--principal=kudu/127.22.37.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.37.3:34609
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.37.3
--webserver_port=43507
--tserver_master_addrs=127.22.37.62:37219
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.37.3
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:38:57.758960 4936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:57.760308 4936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:57.774073 4941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:57.773988 4942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:57.774037 4944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:57.774884 4936 server_base.cc:1034] running on GCE node
I20260620 18:38:57.775501 4936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:57.776453 4936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:57.777671 4936 hybrid_clock.cc:648] HybridClock initialized: now 1781980737777634 us; error 60 us; skew 500 ppm
I20260620 18:38:57.777997 4936 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:38:57 dist-test-slave-5lgp krb5kdc[4031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980737, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:57.781185 4936 init.cc:377] Logged in from keytab as kudu/127.22.37.3@KRBTEST.COM (short username kudu)
I20260620 18:38:57.782598 4936 webserver.cc:460] Webserver started at http://127.22.37.3:43507/ using document root <none> and password file <none>
I20260620 18:38:57.783277 4936 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:57.783440 4936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:57.788794 4936 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 18:38:57.791621 4951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:57.792932 4936 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:38:57.793110 4936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-2/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-2/wal
uuid: "113f3aa889864bd291885fb8a53e7f79"
format_stamp: "Formatted at 2026-06-20 18:38:55 on dist-test-slave-5lgp"
server_key: "e9f76a09726ed0c1e0a9d5bdca8d083f"
server_key_iv: "53f9ff7ba04381dfa98bf4cbf99fa2c0"
server_key_version: "encryptionkey@0"
I20260620 18:38:57.794175 4936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:57.823733 4936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:57.827929 4936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:57.828226 4936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:57.829200 4936 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:38:57.830520 4936 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:38:57.830646 4936 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:57.830785 4936 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:38:57.830873 4936 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:57.863842 4936 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.3:34609
I20260620 18:38:57.863926 5063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.3:34609 every 8 connection(s)
I20260620 18:38:57.865274 4936 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-2/data/info.pb
I20260620 18:38:57.866225 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 4936
Jun 20 18:38:57 dist-test-slave-5lgp krb5kdc[4031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980737, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.3@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM
I20260620 18:38:57.882473 4803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:57.867348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51484 (local address 127.22.37.62:37219)
0620 18:38:57.867612 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:57.867618 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:57.868022 (+ 404us) server_negotiation.cc:407] Connection header received
0620 18:38:57.868233 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:57.868245 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:57.868349 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:57.868466 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:57.870023 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:57.870856 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:57.872424 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:57.872851 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:57.872987 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:57.875967 (+ 2980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:57.875974 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:57.875978 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:57.876021 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:57.879403 (+ 3382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:57.880154 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:57.880160 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:57.880164 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:57.880306 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:57.880678 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:57.880683 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:57.880687 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:57.881197 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:57.881508 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:57.881825 (+ 317us) server_negotiation.cc:299] Negotiation successful
0620 18:38:57.882072 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 18:38:57.886982 5067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:57.868026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.3:37749 (local address 127.22.37.62:37219)
0620 18:38:57.868496 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:57.868502 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:57.869719 (+ 1217us) server_negotiation.cc:407] Connection header received
0620 18:38:57.871592 (+ 1873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:57.871598 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:57.871669 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:57.871775 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:57.873275 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:57.874421 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:57.876018 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:57.876547 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:57.876682 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:57.880631 (+ 3949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:57.880636 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:57.880640 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:57.880667 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:57.883945 (+ 3278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:57.884670 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:57.884675 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:57.884683 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:57.884786 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:57.885223 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:57.885228 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:57.885231 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:57.885560 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:57.885702 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:57.886360 (+ 658us) server_negotiation.cc:299] Negotiation successful
0620 18:38:57.886579 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":259,"threads_started":1}
I20260620 18:38:57.887984 5068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:57.868745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:37219 (local address 127.22.37.3:37749)
0620 18:38:57.869571 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:57.869614 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:57.871350 (+ 1736us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:57.872047 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:57.872055 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:57.872550 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:57.873115 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:57.873133 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:57.874611 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:57.874615 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:57.875845 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:57.875867 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:57.876803 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:57.876809 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:57.877184 (+ 375us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:57.877834 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:57.877869 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:57.880415 (+ 2546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:57.884085 (+ 3670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:57.884094 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:57.884111 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:57.884549 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:57.884909 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:57.884914 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:57.884918 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:57.885100 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:57.885686 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:57.885694 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:57.886024 (+ 330us) client_negotiation.cc:769] Sending connection context
0620 18:38:57.886291 (+ 267us) client_negotiation.cc:240] Negotiation successful
0620 18:38:57.886855 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":340,"threads_started":1}
I20260620 18:38:57.889309 5064 heartbeater.cc:347] Connected to a master server at 127.22.37.62:37219
I20260620 18:38:57.889699 5064 heartbeater.cc:464] Registering TS with master...
I20260620 18:38:57.890409 5064 heartbeater.cc:511] Master 127.22.37.62:37219 requested a full tablet report, sending...
I20260620 18:38:57.891796 4616 ts_manager.cc:194] Registered new tserver with Master: 113f3aa889864bd291885fb8a53e7f79 (127.22.37.3:34609)
I20260620 18:38:57.892630 4616 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.37.3@KRBTEST.COM'} at 127.22.37.3:37749
I20260620 18:38:57.897127 22676 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:38:57.903791 5067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:57.898096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51492 (local address 127.22.37.62:37219)
0620 18:38:57.898253 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:57.898257 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:57.898790 (+ 533us) server_negotiation.cc:407] Connection header received
0620 18:38:57.899041 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:57.899045 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:57.899170 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:57.899284 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:38:57.899672 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:57.900339 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:57.901915 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:57.902318 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:57.902408 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:57.902906 (+ 498us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:38:57.903353 (+ 447us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:38:57.903453 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:57.903546 (+ 93us) server_negotiation.cc:299] Negotiation successful
0620 18:38:57.903606 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 18:38:57.904619 4616 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:51492:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:38:57.911836 4616 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:38:57.935068 5074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:57.923936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.1:39491 (local address 127.0.0.1:59158)
0620 18:38:57.926027 (+ 2091us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:57.926085 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:57.926324 (+ 239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:57.927406 (+ 1082us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:57.927413 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:57.927442 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:38:57.928019 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:57.928036 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:57.930087 (+ 2051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:57.930094 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:57.932142 (+ 2048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:57.932155 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:57.934057 (+ 1902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:57.934063 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:57.934250 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:57.934308 (+ 58us) client_negotiation.cc:769] Sending connection context
0620 18:38:57.934480 (+ 172us) client_negotiation.cc:240] Negotiation successful
0620 18:38:57.934669 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1738,"mutex_wait_us":36,"thread_start_us":1568,"threads_started":1}
I20260620 18:38:57.935047 5075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:57.924478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59158 (local address 127.22.37.1:39491)
0620 18:38:57.925155 (+ 677us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:57.925167 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:57.926541 (+ 1374us) server_negotiation.cc:407] Connection header received
0620 18:38:57.926656 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:57.926664 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:57.926922 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:57.927135 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:38:57.928254 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:57.929940 (+ 1686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:57.932365 (+ 2425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:57.933858 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:57.933989 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:57.934182 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:57.934534 (+ 352us) server_negotiation.cc:299] Negotiation successful
0620 18:38:57.934699 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":331,"threads_started":1}
I20260620 18:38:57.935266 5076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:57.924626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34380 (local address 127.22.37.3:34609)
0620 18:38:57.925591 (+ 965us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:57.925599 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:57.926851 (+ 1252us) server_negotiation.cc:407] Connection header received
0620 18:38:57.927100 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:57.927108 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:57.927334 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:57.927494 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:38:57.928371 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:57.929937 (+ 1566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:57.932489 (+ 2552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:57.933915 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:57.934069 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:57.934288 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:57.934759 (+ 471us) server_negotiation.cc:299] Negotiation successful
0620 18:38:57.934924 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":793,"thread_start_us":587,"threads_started":1}
I20260620 18:38:57.936180 5077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:57.924047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.3:34609 (local address 127.0.0.1:34380)
0620 18:38:57.926729 (+ 2682us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:57.926746 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:57.926895 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:57.927721 (+ 826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:57.927724 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:57.927742 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:38:57.928190 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:57.928202 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:57.930144 (+ 1942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:57.930148 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:57.932303 (+ 2155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:57.932317 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:57.934352 (+ 2035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:57.934367 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:57.934553 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:57.934586 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 18:38:57.935621 (+ 1035us) client_negotiation.cc:240] Negotiation successful
0620 18:38:57.935787 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2563,"mutex_wait_us":146,"thread_start_us":2269,"threads_started":1}
I20260620 18:38:57.936753 5078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:57.926680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.2:41639 (local address 127.0.0.1:60318)
0620 18:38:57.927203 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:57.927218 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:57.927309 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:57.928732 (+ 1423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:57.928736 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:57.929090 (+ 354us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:38:57.929536 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:57.929549 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:57.931482 (+ 1933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:57.931487 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:57.933642 (+ 2155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:57.933657 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:57.935740 (+ 2083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:57.935745 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:57.935929 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:57.935964 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 18:38:57.936065 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 18:38:57.936217 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":647,"threads_started":1}
I20260620 18:38:57.936828 5079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:57.927137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60318 (local address 127.22.37.2:41639)
0620 18:38:57.928283 (+ 1146us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:57.928295 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:57.928320 (+ 25us) server_negotiation.cc:407] Connection header received
0620 18:38:57.928414 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:57.928418 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:57.928559 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:57.928752 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:38:57.929729 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:57.931313 (+ 1584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:57.933806 (+ 2493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:57.935246 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:57.935394 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:57.935587 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:57.936137 (+ 550us) server_negotiation.cc:299] Negotiation successful
0620 18:38:57.936303 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":996,"thread_start_us":889,"threads_started":1}
I20260620 18:38:57.939263 4736 tablet_service.cc:1467] Processing CreateTablet for tablet 00816837f98241b6b1521de521a4dce3 (DEFAULT_TABLE table=test-table [id=7c652ee60f814ba6aff38b84b9d3614c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:38:57.940464 4736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00816837f98241b6b1521de521a4dce3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:57.940078 4999 tablet_service.cc:1467] Processing CreateTablet for tablet 00816837f98241b6b1521de521a4dce3 (DEFAULT_TABLE table=test-table [id=7c652ee60f814ba6aff38b84b9d3614c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:38:57.940404 4868 tablet_service.cc:1467] Processing CreateTablet for tablet 00816837f98241b6b1521de521a4dce3 (DEFAULT_TABLE table=test-table [id=7c652ee60f814ba6aff38b84b9d3614c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:38:57.941479 4999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00816837f98241b6b1521de521a4dce3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:57.941606 4868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00816837f98241b6b1521de521a4dce3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:57.960819 5080 tablet_bootstrap.cc:492] T 00816837f98241b6b1521de521a4dce3 P aa1a40a28b54480c80bbf29fe96919a7: Bootstrap starting.
I20260620 18:38:57.962177 5081 tablet_bootstrap.cc:492] T 00816837f98241b6b1521de521a4dce3 P 113f3aa889864bd291885fb8a53e7f79: Bootstrap starting.
I20260620 18:38:57.963691 5080 tablet_bootstrap.cc:654] T 00816837f98241b6b1521de521a4dce3 P aa1a40a28b54480c80bbf29fe96919a7: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:57.964956 5082 tablet_bootstrap.cc:492] T 00816837f98241b6b1521de521a4dce3 P a50c31a3297f4a65885627657f513f8b: Bootstrap starting.
I20260620 18:38:57.965433 5080 log.cc:824] T 00816837f98241b6b1521de521a4dce3 P aa1a40a28b54480c80bbf29fe96919a7: Log is configured to *not* fsync() on all Append() calls
I20260620 18:38:57.966113 5081 tablet_bootstrap.cc:654] T 00816837f98241b6b1521de521a4dce3 P 113f3aa889864bd291885fb8a53e7f79: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:57.968176 5081 log.cc:824] T 00816837f98241b6b1521de521a4dce3 P 113f3aa889864bd291885fb8a53e7f79: Log is configured to *not* fsync() on all Append() calls
I20260620 18:38:57.969466 5082 tablet_bootstrap.cc:654] T 00816837f98241b6b1521de521a4dce3 P a50c31a3297f4a65885627657f513f8b: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:57.970546 5080 tablet_bootstrap.cc:492] T 00816837f98241b6b1521de521a4dce3 P aa1a40a28b54480c80bbf29fe96919a7: No bootstrap required, opened a new log
I20260620 18:38:57.970840 5080 ts_tablet_manager.cc:1397] T 00816837f98241b6b1521de521a4dce3 P aa1a40a28b54480c80bbf29fe96919a7: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260620 18:38:57.970835 5082 log.cc:824] T 00816837f98241b6b1521de521a4dce3 P a50c31a3297f4a65885627657f513f8b: Log is configured to *not* fsync() on all Append() calls
I20260620 18:38:57.974252 5082 tablet_bootstrap.cc:492] T 00816837f98241b6b1521de521a4dce3 P a50c31a3297f4a65885627657f513f8b: No bootstrap required, opened a new log
I20260620 18:38:57.974568 5082 ts_tablet_manager.cc:1397] T 00816837f98241b6b1521de521a4dce3 P a50c31a3297f4a65885627657f513f8b: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20260620 18:38:57.976852 5081 tablet_bootstrap.cc:492] T 00816837f98241b6b1521de521a4dce3 P 113f3aa889864bd291885fb8a53e7f79: No bootstrap required, opened a new log
I20260620 18:38:57.977105 5081 ts_tablet_manager.cc:1397] T 00816837f98241b6b1521de521a4dce3 P 113f3aa889864bd291885fb8a53e7f79: Time spent bootstrapping tablet: real 0.015s user 0.011s sys 0.000s
I20260620 18:38:57.978993 5080 raft_consensus.cc:348] T 00816837f98241b6b1521de521a4dce3 P aa1a40a28b54480c80bbf29fe96919a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "113f3aa889864bd291885fb8a53e7f79" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 34609 } } peers { permanent_uuid: "aa1a40a28b54480c80bbf29fe96919a7" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 39491 } } peers { permanent_uuid: "a50c31a3297f4a65885627657f513f8b" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 41639 } }
I20260620 18:38:57.979506 5080 raft_consensus.cc:374] T 00816837f98241b6b1521de521a4dce3 P aa1a40a28b54480c80bbf29fe96919a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:57.979641 5080 raft_consensus.cc:729] T 00816837f98241b6b1521de521a4dce3 P aa1a40a28b54480c80bbf29fe96919a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa1a40a28b54480c80bbf29fe96919a7, State: Initialized, Role: FOLLOWER
I20260620 18:38:57.980302 5080 consensus_queue.cc:260] T 00816837f98241b6b1521de521a4dce3 P aa1a40a28b54480c80bbf29fe96919a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "113f3aa889864bd291885fb8a53e7f79" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 34609 } } peers { permanent_uuid: "aa1a40a28b54480c80bbf29fe96919a7" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 39491 } } peers { permanent_uuid: "a50c31a3297f4a65885627657f513f8b" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 41639 } }
I20260620 18:38:57.981335 4801 heartbeater.cc:503] Master 127.22.37.62:37219 was elected leader, sending a full tablet report...
I20260620 18:38:57.982043 5080 ts_tablet_manager.cc:1428] T 00816837f98241b6b1521de521a4dce3 P aa1a40a28b54480c80bbf29fe96919a7: Time spent starting tablet: real 0.011s user 0.006s sys 0.004s
I20260620 18:38:57.985253 5082 raft_consensus.cc:348] T 00816837f98241b6b1521de521a4dce3 P a50c31a3297f4a65885627657f513f8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "113f3aa889864bd291885fb8a53e7f79" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 34609 } } peers { permanent_uuid: "aa1a40a28b54480c80bbf29fe96919a7" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 39491 } } peers { permanent_uuid: "a50c31a3297f4a65885627657f513f8b" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 41639 } }
I20260620 18:38:57.985742 5082 raft_consensus.cc:374] T 00816837f98241b6b1521de521a4dce3 P a50c31a3297f4a65885627657f513f8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:57.985844 5082 raft_consensus.cc:729] T 00816837f98241b6b1521de521a4dce3 P a50c31a3297f4a65885627657f513f8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a50c31a3297f4a65885627657f513f8b, State: Initialized, Role: FOLLOWER
I20260620 18:38:57.986486 5082 consensus_queue.cc:260] T 00816837f98241b6b1521de521a4dce3 P a50c31a3297f4a65885627657f513f8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "113f3aa889864bd291885fb8a53e7f79" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 34609 } } peers { permanent_uuid: "aa1a40a28b54480c80bbf29fe96919a7" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 39491 } } peers { permanent_uuid: "a50c31a3297f4a65885627657f513f8b" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 41639 } }
I20260620 18:38:57.987524 4933 heartbeater.cc:503] Master 127.22.37.62:37219 was elected leader, sending a full tablet report...
I20260620 18:38:57.987692 5081 raft_consensus.cc:348] T 00816837f98241b6b1521de521a4dce3 P 113f3aa889864bd291885fb8a53e7f79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "113f3aa889864bd291885fb8a53e7f79" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 34609 } } peers { permanent_uuid: "aa1a40a28b54480c80bbf29fe96919a7" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 39491 } } peers { permanent_uuid: "a50c31a3297f4a65885627657f513f8b" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 41639 } }
I20260620 18:38:57.988113 5081 raft_consensus.cc:374] T 00816837f98241b6b1521de521a4dce3 P 113f3aa889864bd291885fb8a53e7f79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:57.988236 5082 ts_tablet_manager.cc:1428] T 00816837f98241b6b1521de521a4dce3 P a50c31a3297f4a65885627657f513f8b: Time spent starting tablet: real 0.014s user 0.005s sys 0.007s
I20260620 18:38:57.988219 5081 raft_consensus.cc:729] T 00816837f98241b6b1521de521a4dce3 P 113f3aa889864bd291885fb8a53e7f79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 113f3aa889864bd291885fb8a53e7f79, State: Initialized, Role: FOLLOWER
I20260620 18:38:57.988773 5081 consensus_queue.cc:260] T 00816837f98241b6b1521de521a4dce3 P 113f3aa889864bd291885fb8a53e7f79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "113f3aa889864bd291885fb8a53e7f79" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 34609 } } peers { permanent_uuid: "aa1a40a28b54480c80bbf29fe96919a7" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 39491 } } peers { permanent_uuid: "a50c31a3297f4a65885627657f513f8b" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 41639 } }
I20260620 18:38:57.989868 5064 heartbeater.cc:503] Master 127.22.37.62:37219 was elected leader, sending a full tablet report...
I20260620 18:38:57.990414 5081 ts_tablet_manager.cc:1428] T 00816837f98241b6b1521de521a4dce3 P 113f3aa889864bd291885fb8a53e7f79: Time spent starting tablet: real 0.013s user 0.010s sys 0.004s
W20260620 18:38:58.117257 5065 tablet.cc:2369] T 00816837f98241b6b1521de521a4dce3 P 113f3aa889864bd291885fb8a53e7f79: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:38:58.152911 4934 tablet.cc:2369] T 00816837f98241b6b1521de521a4dce3 P a50c31a3297f4a65885627657f513f8b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:38:58.200786 4802 tablet.cc:2369] T 00816837f98241b6b1521de521a4dce3 P aa1a40a28b54480c80bbf29fe96919a7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:38:58.348623 5086 raft_consensus.cc:482] T 00816837f98241b6b1521de521a4dce3 P aa1a40a28b54480c80bbf29fe96919a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:38:58.348974 5086 raft_consensus.cc:504] T 00816837f98241b6b1521de521a4dce3 P aa1a40a28b54480c80bbf29fe96919a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "113f3aa889864bd291885fb8a53e7f79" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 34609 } } peers { permanent_uuid: "aa1a40a28b54480c80bbf29fe96919a7" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 39491 } } peers { permanent_uuid: "a50c31a3297f4a65885627657f513f8b" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 41639 } }
I20260620 18:38:58.350832 5086 leader_election.cc:288] T 00816837f98241b6b1521de521a4dce3 P aa1a40a28b54480c80bbf29fe96919a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 113f3aa889864bd291885fb8a53e7f79 (127.22.37.3:34609), a50c31a3297f4a65885627657f513f8b (127.22.37.2:41639)
I20260620 18:38:58.357743 5076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:58.351264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.1:59825 (local address 127.22.37.3:34609)
0620 18:38:58.351458 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:58.351463 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:58.352035 (+ 572us) server_negotiation.cc:407] Connection header received
0620 18:38:58.352300 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:58.352309 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:58.352408 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:58.352519 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:38:58.353336 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:58.354304 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:58.356049 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:58.356798 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:58.356953 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:58.357062 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:58.357328 (+ 266us) server_negotiation.cc:299] Negotiation successful
0620 18:38:58.357480 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 18:38:58.357921 5089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:58.351252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.3:34609 (local address 127.22.37.1:59825)
0620 18:38:58.351907 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:58.351922 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:58.352107 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:58.352576 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:58.352582 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:58.352607 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:38:58.353145 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:58.353161 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:58.354433 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:58.354438 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:58.355884 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:58.355895 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:58.357006 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:58.357012 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:58.357169 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:58.357192 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 18:38:58.357301 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 18:38:58.357466 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"spinlock_wait_cycles":25984,"thread_start_us":365,"threads_started":1}
I20260620 18:38:58.358177 5079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:58.352364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.1:57985 (local address 127.22.37.2:41639)
0620 18:38:58.352576 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:58.352582 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:58.352776 (+ 194us) server_negotiation.cc:407] Connection header received
0620 18:38:58.353075 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:58.353083 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:58.353230 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:58.353324 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:38:58.354047 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:58.354735 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:58.356402 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:58.357216 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:58.357402 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:58.357505 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:58.357805 (+ 300us) server_negotiation.cc:299] Negotiation successful
0620 18:38:58.357925 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 18:38:58.358206 5088 raft_consensus.cc:482] T 00816837f98241b6b1521de521a4dce3 P 113f3aa889864bd291885fb8a53e7f79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:38:58.358515 5088 raft_consensus.cc:504] T 00816837f98241b6b1521de521a4dce3 P 113f3aa889864bd291885fb8a53e7f79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "113f3aa889864bd291885fb8a53e7f79" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 34609 } } peers { permanent_uuid: "aa1a40a28b54480c80bbf29fe96919a7" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 39491 } } peers { permanent_uuid: "a50c31a3297f4a65885627657f513f8b" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 41639 } }
I20260620 18:38:58.358894 5090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:58.352064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.2:41639 (local address 127.22.37.1:57985)
0620 18:38:58.352658 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:58.352674 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:58.352832 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:58.353356 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:58.353360 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:58.353426 (+ 66us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:38:58.353880 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:58.353896 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:58.354882 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:58.354890 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:58.356238 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:58.356248 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:58.357476 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:58.357483 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:58.357648 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:58.357671 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 18:38:58.357780 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 18:38:58.358579 (+ 799us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":282,"threads_started":1}
I20260620 18:38:58.358995 5019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00816837f98241b6b1521de521a4dce3" candidate_uuid: "aa1a40a28b54480c80bbf29fe96919a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "113f3aa889864bd291885fb8a53e7f79" is_pre_election: true
I20260620 18:38:58.359520 5019 raft_consensus.cc:2454] T 00816837f98241b6b1521de521a4dce3 P 113f3aa889864bd291885fb8a53e7f79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa1a40a28b54480c80bbf29fe96919a7 in term 0.
I20260620 18:38:58.360015 4888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00816837f98241b6b1521de521a4dce3" candidate_uuid: "aa1a40a28b54480c80bbf29fe96919a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a50c31a3297f4a65885627657f513f8b" is_pre_election: true
I20260620 18:38:58.360252 4690 leader_election.cc:302] T 00816837f98241b6b1521de521a4dce3 P aa1a40a28b54480c80bbf29fe96919a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 113f3aa889864bd291885fb8a53e7f79, aa1a40a28b54480c80bbf29fe96919a7; no voters:
I20260620 18:38:58.360529 5088 leader_election.cc:288] T 00816837f98241b6b1521de521a4dce3 P 113f3aa889864bd291885fb8a53e7f79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aa1a40a28b54480c80bbf29fe96919a7 (127.22.37.1:39491), a50c31a3297f4a65885627657f513f8b (127.22.37.2:41639)
I20260620 18:38:58.360555 4888 raft_consensus.cc:2454] T 00816837f98241b6b1521de521a4dce3 P a50c31a3297f4a65885627657f513f8b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa1a40a28b54480c80bbf29fe96919a7 in term 0.
I20260620 18:38:58.360715 5086 raft_consensus.cc:2781] T 00816837f98241b6b1521de521a4dce3 P aa1a40a28b54480c80bbf29fe96919a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:38:58.360859 5086 raft_consensus.cc:482] T 00816837f98241b6b1521de521a4dce3 P aa1a40a28b54480c80bbf29fe96919a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:38:58.360981 5086 raft_consensus.cc:3037] T 00816837f98241b6b1521de521a4dce3 P aa1a40a28b54480c80bbf29fe96919a7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:58.363734 5086 raft_consensus.cc:504] T 00816837f98241b6b1521de521a4dce3 P aa1a40a28b54480c80bbf29fe96919a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "113f3aa889864bd291885fb8a53e7f79" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 34609 } } peers { permanent_uuid: "aa1a40a28b54480c80bbf29fe96919a7" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 39491 } } peers { permanent_uuid: "a50c31a3297f4a65885627657f513f8b" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 41639 } }
I20260620 18:38:58.364710 5086 leader_election.cc:288] T 00816837f98241b6b1521de521a4dce3 P aa1a40a28b54480c80bbf29fe96919a7 [CANDIDATE]: Term 1 election: Requested vote from peers 113f3aa889864bd291885fb8a53e7f79 (127.22.37.3:34609), a50c31a3297f4a65885627657f513f8b (127.22.37.2:41639)
I20260620 18:38:58.364784 5019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00816837f98241b6b1521de521a4dce3" candidate_uuid: "aa1a40a28b54480c80bbf29fe96919a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "113f3aa889864bd291885fb8a53e7f79"
I20260620 18:38:58.365010 5019 raft_consensus.cc:3037] T 00816837f98241b6b1521de521a4dce3 P 113f3aa889864bd291885fb8a53e7f79 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:58.365572 4888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00816837f98241b6b1521de521a4dce3" candidate_uuid: "aa1a40a28b54480c80bbf29fe96919a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a50c31a3297f4a65885627657f513f8b"
I20260620 18:38:58.365794 4888 raft_consensus.cc:3037] T 00816837f98241b6b1521de521a4dce3 P a50c31a3297f4a65885627657f513f8b [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:58.368407 5019 raft_consensus.cc:2454] T 00816837f98241b6b1521de521a4dce3 P 113f3aa889864bd291885fb8a53e7f79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa1a40a28b54480c80bbf29fe96919a7 in term 1.
I20260620 18:38:58.368887 4690 leader_election.cc:302] T 00816837f98241b6b1521de521a4dce3 P aa1a40a28b54480c80bbf29fe96919a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 113f3aa889864bd291885fb8a53e7f79, aa1a40a28b54480c80bbf29fe96919a7; no voters:
I20260620 18:38:58.368996 5068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:58.360629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.1:39491 (local address 127.22.37.3:40515)
0620 18:38:58.361006 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:58.361024 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:58.361122 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:58.362048 (+ 926us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:58.362055 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:58.362086 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:38:58.362266 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:58.362276 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:58.364120 (+ 1844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:58.364126 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:58.366346 (+ 2220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:58.366373 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:58.368124 (+ 1751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:58.368137 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:58.368357 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:58.368414 (+ 57us) client_negotiation.cc:769] Sending connection context
0620 18:38:58.368526 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 18:38:58.368650 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227}
I20260620 18:38:58.369251 4888 raft_consensus.cc:2454] T 00816837f98241b6b1521de521a4dce3 P a50c31a3297f4a65885627657f513f8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa1a40a28b54480c80bbf29fe96919a7 in term 1.
I20260620 18:38:58.369392 5086 raft_consensus.cc:2781] T 00816837f98241b6b1521de521a4dce3 P aa1a40a28b54480c80bbf29fe96919a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:38:58.369747 5075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:58.361465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.3:40515 (local address 127.22.37.1:39491)
0620 18:38:58.361646 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:58.361656 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:58.361672 (+ 16us) server_negotiation.cc:407] Connection header received
0620 18:38:58.361721 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:58.361726 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:58.361803 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:58.362036 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:38:58.362449 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:58.363893 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:58.366548 (+ 2655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:58.367778 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:58.367895 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:58.367981 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:58.369222 (+ 1241us) server_negotiation.cc:299] Negotiation successful
0620 18:38:58.369352 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 18:38:58.370582 4756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00816837f98241b6b1521de521a4dce3" candidate_uuid: "113f3aa889864bd291885fb8a53e7f79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa1a40a28b54480c80bbf29fe96919a7" is_pre_election: true
I20260620 18:38:58.371027 5086 raft_consensus.cc:686] T 00816837f98241b6b1521de521a4dce3 P aa1a40a28b54480c80bbf29fe96919a7 [term 1 LEADER]: Becoming Leader. State: Replica: aa1a40a28b54480c80bbf29fe96919a7, State: Running, Role: LEADER
I20260620 18:38:58.371483 5086 consensus_queue.cc:237] T 00816837f98241b6b1521de521a4dce3 P aa1a40a28b54480c80bbf29fe96919a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "113f3aa889864bd291885fb8a53e7f79" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 34609 } } peers { permanent_uuid: "aa1a40a28b54480c80bbf29fe96919a7" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 39491 } } peers { permanent_uuid: "a50c31a3297f4a65885627657f513f8b" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 41639 } }
I20260620 18:38:58.371922 5079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:58.361500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.3:38231 (local address 127.22.37.2:41639)
0620 18:38:58.362036 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:58.362042 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:58.364584 (+ 2542us) server_negotiation.cc:407] Connection header received
0620 18:38:58.364861 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:58.364867 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:58.364938 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:58.365001 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:38:58.365972 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:58.367325 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:58.370100 (+ 2775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:58.371071 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:58.371215 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:58.371308 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:58.371565 (+ 257us) server_negotiation.cc:299] Negotiation successful
0620 18:38:58.371683 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329}
I20260620 18:38:58.371961 5091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:58.361057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.2:41639 (local address 127.22.37.3:38231)
0620 18:38:58.364491 (+ 3434us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:58.364513 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:58.364692 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:58.365239 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:58.365245 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:58.365269 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:38:58.365759 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:58.365778 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:58.367501 (+ 1723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:58.367506 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:58.369929 (+ 2423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:58.369944 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:58.371264 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:58.371269 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:58.371413 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:58.371434 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 18:38:58.371529 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 18:38:58.371643 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3281,"thread_start_us":3040,"threads_started":1}
I20260620 18:38:58.372525 4888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00816837f98241b6b1521de521a4dce3" candidate_uuid: "113f3aa889864bd291885fb8a53e7f79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a50c31a3297f4a65885627657f513f8b" is_pre_election: true
I20260620 18:38:58.372784 4888 raft_consensus.cc:2379] T 00816837f98241b6b1521de521a4dce3 P a50c31a3297f4a65885627657f513f8b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 113f3aa889864bd291885fb8a53e7f79 in current term 1: Already voted for candidate aa1a40a28b54480c80bbf29fe96919a7 in this term.
I20260620 18:38:58.375985 4952 leader_election.cc:302] T 00816837f98241b6b1521de521a4dce3 P 113f3aa889864bd291885fb8a53e7f79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 113f3aa889864bd291885fb8a53e7f79; no voters: a50c31a3297f4a65885627657f513f8b, aa1a40a28b54480c80bbf29fe96919a7
I20260620 18:38:58.376407 5088 raft_consensus.cc:2726] T 00816837f98241b6b1521de521a4dce3 P 113f3aa889864bd291885fb8a53e7f79 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 18:38:58.376899 4616 catalog_manager.cc:5495] T 00816837f98241b6b1521de521a4dce3 P aa1a40a28b54480c80bbf29fe96919a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to aa1a40a28b54480c80bbf29fe96919a7 (127.22.37.1). New cstate: current_term: 1 leader_uuid: "aa1a40a28b54480c80bbf29fe96919a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "113f3aa889864bd291885fb8a53e7f79" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 34609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aa1a40a28b54480c80bbf29fe96919a7" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 39491 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a50c31a3297f4a65885627657f513f8b" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 41639 } health_report { overall_health: UNKNOWN } } }
I20260620 18:38:58.395020 5075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:58.388602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49970 (local address 127.22.37.1:39491)
0620 18:38:58.388795 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:58.388801 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:58.388860 (+ 59us) server_negotiation.cc:407] Connection header received
0620 18:38:58.389089 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:58.389093 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:58.389183 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:58.389344 (+ 161us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:38:58.389783 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:58.390592 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:58.392772 (+ 2180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:58.393180 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:58.393355 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:58.393887 (+ 532us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:38:58.394404 (+ 517us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:38:58.394577 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:58.394637 (+ 60us) server_negotiation.cc:299] Negotiation successful
0620 18:38:58.394727 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 18:38:58.403481 5019 raft_consensus.cc:1261] T 00816837f98241b6b1521de521a4dce3 P 113f3aa889864bd291885fb8a53e7f79 [term 1 FOLLOWER]: Refusing update from remote peer aa1a40a28b54480c80bbf29fe96919a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:38:58.403481 4888 raft_consensus.cc:1261] T 00816837f98241b6b1521de521a4dce3 P a50c31a3297f4a65885627657f513f8b [term 1 FOLLOWER]: Refusing update from remote peer aa1a40a28b54480c80bbf29fe96919a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:38:58.404405 5092 consensus_queue.cc:1035] T 00816837f98241b6b1521de521a4dce3 P aa1a40a28b54480c80bbf29fe96919a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "113f3aa889864bd291885fb8a53e7f79" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 34609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:38:58.404762 5086 consensus_queue.cc:1035] T 00816837f98241b6b1521de521a4dce3 P aa1a40a28b54480c80bbf29fe96919a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a50c31a3297f4a65885627657f513f8b" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:38:58.420126 5095 mvcc.cc:205] Tried to move back new op lower bound from 7298993104490160128 to 7298993104374661120. Current Snapshot: MvccSnapshot[applied={T|T < 7298993104490160128}]
I20260620 18:38:58.433853 4616 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:51492:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:38:58.434109 4616 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:51492:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:38:58.437404 4616 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b7f7b758d94d44a19e5997546da74de4: Sending DeleteTablet for 3 replicas of tablet 00816837f98241b6b1521de521a4dce3
I20260620 18:38:58.438578 4999 tablet_service.cc:1514] Processing DeleteTablet for tablet 00816837f98241b6b1521de521a4dce3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:38:58 UTC) from {username='kudu', principal='kudu/127.22.37.62@KRBTEST.COM'} at 127.0.0.1:34380
I20260620 18:38:58.438652 4736 tablet_service.cc:1514] Processing DeleteTablet for tablet 00816837f98241b6b1521de521a4dce3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:38:58 UTC) from {username='kudu', principal='kudu/127.22.37.62@KRBTEST.COM'} at 127.0.0.1:59158
I20260620 18:38:58.438992 4868 tablet_service.cc:1514] Processing DeleteTablet for tablet 00816837f98241b6b1521de521a4dce3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:38:58 UTC) from {username='kudu', principal='kudu/127.22.37.62@KRBTEST.COM'} at 127.0.0.1:60318
I20260620 18:38:58.439989 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 4653
I20260620 18:38:58.450006 5106 tablet_replica.cc:330] T 00816837f98241b6b1521de521a4dce3 P aa1a40a28b54480c80bbf29fe96919a7: stopping tablet replica
I20260620 18:38:58.450510 5106 raft_consensus.cc:2229] T 00816837f98241b6b1521de521a4dce3 P aa1a40a28b54480c80bbf29fe96919a7 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:38:58.452066 5105 tablet_replica.cc:330] T 00816837f98241b6b1521de521a4dce3 P 113f3aa889864bd291885fb8a53e7f79: stopping tablet replica
I20260620 18:38:58.452504 5105 raft_consensus.cc:2229] T 00816837f98241b6b1521de521a4dce3 P 113f3aa889864bd291885fb8a53e7f79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:38:58.453001 5105 raft_consensus.cc:2258] T 00816837f98241b6b1521de521a4dce3 P 113f3aa889864bd291885fb8a53e7f79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:38:58.454615 5107 tablet_replica.cc:330] T 00816837f98241b6b1521de521a4dce3 P a50c31a3297f4a65885627657f513f8b: stopping tablet replica
I20260620 18:38:58.454875 5106 raft_consensus.cc:2258] T 00816837f98241b6b1521de521a4dce3 P aa1a40a28b54480c80bbf29fe96919a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:38:58.455036 5107 raft_consensus.cc:2229] T 00816837f98241b6b1521de521a4dce3 P a50c31a3297f4a65885627657f513f8b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:38:58.455570 5107 raft_consensus.cc:2258] T 00816837f98241b6b1521de521a4dce3 P a50c31a3297f4a65885627657f513f8b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:38:58.458089 5107 ts_tablet_manager.cc:1905] T 00816837f98241b6b1521de521a4dce3 P a50c31a3297f4a65885627657f513f8b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:38:58.458758 5106 ts_tablet_manager.cc:1905] T 00816837f98241b6b1521de521a4dce3 P aa1a40a28b54480c80bbf29fe96919a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:38:58.460055 5105 ts_tablet_manager.cc:1905] T 00816837f98241b6b1521de521a4dce3 P 113f3aa889864bd291885fb8a53e7f79: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:38:58.466951 5105 ts_tablet_manager.cc:1918] T 00816837f98241b6b1521de521a4dce3 P 113f3aa889864bd291885fb8a53e7f79: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:38:58.468402 5105 log.cc:1196] T 00816837f98241b6b1521de521a4dce3 P 113f3aa889864bd291885fb8a53e7f79: Deleting WAL directory at /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-2/wal/wals/00816837f98241b6b1521de521a4dce3
I20260620 18:38:58.469089 5105 ts_tablet_manager.cc:1939] T 00816837f98241b6b1521de521a4dce3 P 113f3aa889864bd291885fb8a53e7f79: Deleting consensus metadata
I20260620 18:38:58.471300 4601 catalog_manager.cc:4841] TS 113f3aa889864bd291885fb8a53e7f79 (127.22.37.3:34609): tablet 00816837f98241b6b1521de521a4dce3 (table test-table [id=7c652ee60f814ba6aff38b84b9d3614c]) successfully deleted
I20260620 18:38:58.474051 5107 ts_tablet_manager.cc:1918] T 00816837f98241b6b1521de521a4dce3 P a50c31a3297f4a65885627657f513f8b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:38:58.474092 5106 ts_tablet_manager.cc:1918] T 00816837f98241b6b1521de521a4dce3 P aa1a40a28b54480c80bbf29fe96919a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:38:58.474351 5106 log.cc:1196] T 00816837f98241b6b1521de521a4dce3 P aa1a40a28b54480c80bbf29fe96919a7: Deleting WAL directory at /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-0/wal/wals/00816837f98241b6b1521de521a4dce3
I20260620 18:38:58.474570 5107 log.cc:1196] T 00816837f98241b6b1521de521a4dce3 P a50c31a3297f4a65885627657f513f8b: Deleting WAL directory at /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-1/wal/wals/00816837f98241b6b1521de521a4dce3
I20260620 18:38:58.475217 5107 ts_tablet_manager.cc:1939] T 00816837f98241b6b1521de521a4dce3 P a50c31a3297f4a65885627657f513f8b: Deleting consensus metadata
I20260620 18:38:58.475301 5106 ts_tablet_manager.cc:1939] T 00816837f98241b6b1521de521a4dce3 P aa1a40a28b54480c80bbf29fe96919a7: Deleting consensus metadata
I20260620 18:38:58.477149 4600 catalog_manager.cc:4841] TS aa1a40a28b54480c80bbf29fe96919a7 (127.22.37.1:39491): tablet 00816837f98241b6b1521de521a4dce3 (table test-table [id=7c652ee60f814ba6aff38b84b9d3614c]) successfully deleted
I20260620 18:38:58.477988 4601 catalog_manager.cc:4841] TS a50c31a3297f4a65885627657f513f8b (127.22.37.2:41639): tablet 00816837f98241b6b1521de521a4dce3 (table test-table [id=7c652ee60f814ba6aff38b84b9d3614c]) successfully deleted
I20260620 18:38:58.483050 5075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:58.440896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49986 (local address 127.22.37.1:39491)
0620 18:38:58.441147 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:58.441157 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:58.453935 (+ 12778us) server_negotiation.cc:407] Connection header received
0620 18:38:58.454145 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:58.454153 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:58.454268 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:58.454377 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:58.456823 (+ 2446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:58.457971 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:58.467250 (+ 9279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:58.467909 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:58.468043 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:58.473534 (+ 5491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:58.473553 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:58.473571 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:58.473625 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:58.479997 (+ 6372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:58.480798 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:58.480810 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:58.480818 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:58.480979 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:58.481467 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:58.481471 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:58.481473 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:58.481852 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:58.482165 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:58.482486 (+ 321us) server_negotiation.cc:299] Negotiation successful
0620 18:38:58.482734 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 18:38:58.483726 4796 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:38:58.585450 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 4653
I20260620 18:38:58.604398 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 4805
I20260620 18:38:58.618090 5079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:58.605195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49658 (local address 127.22.37.2:41639)
0620 18:38:58.605469 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:58.605473 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:58.605492 (+ 19us) server_negotiation.cc:407] Connection header received
0620 18:38:58.605603 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:58.605607 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:58.605668 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:58.605750 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:58.607097 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:58.607779 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:58.609102 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:58.609648 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:58.609819 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:58.612752 (+ 2933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:58.612766 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:58.612782 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:58.612816 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:58.615635 (+ 2819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:58.616247 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:58.616254 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:58.616256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:58.616425 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:58.616735 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:58.616738 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:58.616740 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:58.617056 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:58.617271 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:58.617593 (+ 322us) server_negotiation.cc:299] Negotiation successful
0620 18:38:58.617752 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 18:38:58.618680 4928 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:38:58.709681 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 4805
I20260620 18:38:58.727073 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 4936
I20260620 18:38:58.741410 5076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:58.727853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45194 (local address 127.22.37.3:34609)
0620 18:38:58.728055 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:58.728061 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:58.728080 (+ 19us) server_negotiation.cc:407] Connection header received
0620 18:38:58.728267 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:58.728273 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:58.728350 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:58.728468 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:58.730007 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:58.731017 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:58.732434 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:58.732856 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:58.733000 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:58.735677 (+ 2677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:58.735703 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:58.735719 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:58.735757 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:58.738619 (+ 2862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:58.739308 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:58.739316 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:58.739319 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:58.739460 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:58.739827 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:58.739833 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:58.739836 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:58.740183 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:58.740483 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:58.740768 (+ 285us) server_negotiation.cc:299] Negotiation successful
0620 18:38:58.741015 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 18:38:58.742090 5059 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:38:58.832086 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 4936
I20260620 18:38:58.848680 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 4583
I20260620 18:38:58.849893 4646 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:38:58.968865 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 4583
I20260620 18:38:58.998051 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.62:37219
--webserver_interface=127.22.37.62
--webserver_port=36715
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab
--principal=kudu/127.22.37.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:36803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.37.62:37219
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:38:59.338498 5113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:59.338846 5113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:59.339015 5113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:59.346517 5113 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:38:59.346661 5113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:59.346771 5113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:59.346874 5113 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:38:59.346956 5113 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:38:59.356828 5113 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:36803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.37.62:37219
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab
--principal=kudu/127.22.37.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.62:37219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.37.62
--webserver_port=36715
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:38:59.358201 5113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:59.359524 5113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:59.371294 5118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:59.371239 5121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:59.371187 5119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:59.372128 5113 server_base.cc:1034] running on GCE node
I20260620 18:38:59.372964 5113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:59.374542 5113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:59.375785 5113 hybrid_clock.cc:648] HybridClock initialized: now 1781980739375733 us; error 59 us; skew 500 ppm
I20260620 18:38:59.376139 5113 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:38:59 dist-test-slave-5lgp krb5kdc[4031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980739, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:59.379309 5113 init.cc:377] Logged in from keytab as kudu/127.22.37.62@KRBTEST.COM (short username kudu)
I20260620 18:38:59.380800 5113 webserver.cc:460] Webserver started at http://127.22.37.62:36715/ using document root <none> and password file <none>
I20260620 18:38:59.381561 5113 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:59.381671 5113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:59.387215 5113 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 18:38:59.390162 5128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:59.391552 5113 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:38:59.391739 5113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/master-0/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/master-0/wal
uuid: "b7f7b758d94d44a19e5997546da74de4"
format_stamp: "Formatted at 2026-06-20 18:38:53 on dist-test-slave-5lgp"
server_key: "b1b4626643bf9bc61dadb969b47e777c"
server_key_iv: "d22eee2a99893023ebdce3d2b992ae36"
server_key_version: "encryptionkey@0"
I20260620 18:38:59.392773 5113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:59.420717 5113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:59.424765 5113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:59.425068 5113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:59.444290 5113 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.62:37219
I20260620 18:38:59.444329 5179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.62:37219 every 8 connection(s)
I20260620 18:38:59.445885 5113 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
I20260620 18:38:59.447918 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 5113
I20260620 18:38:59.448961 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.37.1:39491
--local_ip_for_outbound_sockets=127.22.37.1
--tserver_master_addrs=127.22.37.62:37219
--webserver_port=41659
--webserver_interface=127.22.37.1
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.1.keytab
--principal=kudu/127.22.37.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:36803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:38:59.455788 5180 sys_catalog.cc:263] Verifying existing consensus state
I20260620 18:38:59.459656 5180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7f7b758d94d44a19e5997546da74de4: Bootstrap starting.
I20260620 18:38:59.481433 5180 log.cc:824] T 00000000000000000000000000000000 P b7f7b758d94d44a19e5997546da74de4: Log is configured to *not* fsync() on all Append() calls
I20260620 18:38:59.498045 5180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7f7b758d94d44a19e5997546da74de4: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 18:38:59.498571 5180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7f7b758d94d44a19e5997546da74de4: Bootstrap complete.
I20260620 18:38:59.507676 5180 raft_consensus.cc:348] T 00000000000000000000000000000000 P b7f7b758d94d44a19e5997546da74de4 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7f7b758d94d44a19e5997546da74de4" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 37219 } }
I20260620 18:38:59.508555 5180 raft_consensus.cc:729] T 00000000000000000000000000000000 P b7f7b758d94d44a19e5997546da74de4 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7f7b758d94d44a19e5997546da74de4, State: Initialized, Role: FOLLOWER
I20260620 18:38:59.509172 5180 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7f7b758d94d44a19e5997546da74de4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7f7b758d94d44a19e5997546da74de4" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 37219 } }
I20260620 18:38:59.509474 5180 raft_consensus.cc:388] T 00000000000000000000000000000000 P b7f7b758d94d44a19e5997546da74de4 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:38:59.509600 5180 raft_consensus.cc:482] T 00000000000000000000000000000000 P b7f7b758d94d44a19e5997546da74de4 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:38:59.509779 5180 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b7f7b758d94d44a19e5997546da74de4 [term 2 FOLLOWER]: Advancing to term 3
I20260620 18:38:59.513480 5180 raft_consensus.cc:504] T 00000000000000000000000000000000 P b7f7b758d94d44a19e5997546da74de4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7f7b758d94d44a19e5997546da74de4" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 37219 } }
I20260620 18:38:59.513952 5180 leader_election.cc:302] T 00000000000000000000000000000000 P b7f7b758d94d44a19e5997546da74de4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b7f7b758d94d44a19e5997546da74de4; no voters:
I20260620 18:38:59.514653 5180 leader_election.cc:288] T 00000000000000000000000000000000 P b7f7b758d94d44a19e5997546da74de4 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 18:38:59.514786 5184 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b7f7b758d94d44a19e5997546da74de4 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 18:38:59.516117 5184 raft_consensus.cc:686] T 00000000000000000000000000000000 P b7f7b758d94d44a19e5997546da74de4 [term 3 LEADER]: Becoming Leader. State: Replica: b7f7b758d94d44a19e5997546da74de4, State: Running, Role: LEADER
I20260620 18:38:59.516705 5184 consensus_queue.cc:237] T 00000000000000000000000000000000 P b7f7b758d94d44a19e5997546da74de4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7f7b758d94d44a19e5997546da74de4" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 37219 } }
I20260620 18:38:59.516916 5180 sys_catalog.cc:564] T 00000000000000000000000000000000 P b7f7b758d94d44a19e5997546da74de4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:38:59.519336 5186 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7f7b758d94d44a19e5997546da74de4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b7f7b758d94d44a19e5997546da74de4. Latest consensus state: current_term: 3 leader_uuid: "b7f7b758d94d44a19e5997546da74de4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7f7b758d94d44a19e5997546da74de4" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 37219 } } }
I20260620 18:38:59.519779 5186 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7f7b758d94d44a19e5997546da74de4 [sys.catalog]: This master's current role is: LEADER
I20260620 18:38:59.520352 5185 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7f7b758d94d44a19e5997546da74de4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "b7f7b758d94d44a19e5997546da74de4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7f7b758d94d44a19e5997546da74de4" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 37219 } } }
I20260620 18:38:59.520622 5185 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7f7b758d94d44a19e5997546da74de4 [sys.catalog]: This master's current role is: LEADER
I20260620 18:38:59.525360 5191 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:38:59.536167 5191 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:38:59.538301 5191 catalog_manager.cc:1259] Loaded cluster ID: 001289474cef41599d8c617f2c4b8f7b
I20260620 18:38:59.538522 5191 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260620 18:38:59.541460 5191 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f8a69b4b9ad google::LogMessage::Fail() at ??:0
@ 0x7f8a69b4e88a google::LogMessage::SendToLog() at ??:0
@ 0x7f8a69b4b4de google::LogMessage::Flush() at ??:0
@ 0x7f8a69b4ca19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f8a7f57e512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f8a7f57b976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f8a7f603610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f8a7f603439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f8a6c797345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f8a6c79a91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f8a6c79a799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f8a6c775eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f8a6fa4f6db start_thread at ??:0
@ 0x7f8a689e561f clone at ??:0
W20260620 18:38:59.829937 5182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:59.830289 5182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:59.830469 5182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:59.838085 5182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:59.838266 5182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:59.838454 5182 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.37.1
I20260620 18:38:59.848774 5182 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:36803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.1.keytab
--principal=kudu/127.22.37.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.37.1:39491
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.37.1
--webserver_port=41659
--tserver_master_addrs=127.22.37.62:37219
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.37.1
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:38:59.850268 5182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:59.851686 5182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:59.865687 5224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:59.865775 5221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:59.865753 5222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:59.866901 5182 server_base.cc:1034] running on GCE node
I20260620 18:38:59.867545 5182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:59.868599 5182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:59.869825 5182 hybrid_clock.cc:648] HybridClock initialized: now 1781980739869775 us; error 46 us; skew 500 ppm
I20260620 18:38:59.870157 5182 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:38:59 dist-test-slave-5lgp krb5kdc[4031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980739, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:59.873355 5182 init.cc:377] Logged in from keytab as kudu/127.22.37.1@KRBTEST.COM (short username kudu)
I20260620 18:38:59.874917 5182 webserver.cc:460] Webserver started at http://127.22.37.1:41659/ using document root <none> and password file <none>
I20260620 18:38:59.875631 5182 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:59.875782 5182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:59.881258 5182 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 18:38:59.883996 5231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:59.885540 5182 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:38:59.885738 5182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-0/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-0/wal
uuid: "aa1a40a28b54480c80bbf29fe96919a7"
format_stamp: "Formatted at 2026-06-20 18:38:54 on dist-test-slave-5lgp"
server_key: "c42730258bd5fb3f1e73310604833a5a"
server_key_iv: "43c786ff1cc383934b071c646e1298a1"
server_key_version: "encryptionkey@0"
I20260620 18:38:59.886744 5182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:59.901949 5182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:59.906553 5182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:59.906855 5182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:59.907958 5182 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:38:59.909422 5182 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:38:59.909554 5182 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:59.909695 5182 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:38:59.909786 5182 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:59.944051 5182 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.1:39491
I20260620 18:38:59.944141 5343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.1:39491 every 8 connection(s)
I20260620 18:38:59.945644 5182 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-0/data/info.pb
I20260620 18:38:59.950438 5346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:59.948591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:37219 (local address 127.22.37.1:44711)
0620 18:38:59.949282 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:59.949539 (+ 257us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:37219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":264,"threads_started":1}
W20260620 18:38:59.951221 5344 heartbeater.cc:644] Failed to heartbeat to 127.22.37.62:37219 (0 consecutive failures): Network error: Failed to ping master at 127.22.37.62:37219: Client connection negotiation failed: client connection to 127.22.37.62:37219: connect: Connection refused (error 111)
I20260620 18:38:59.952000 5346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:59.951685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:37219 (local address 127.22.37.1:47305)
0620 18:38:59.951818 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:59.951893 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:37219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260620 18:38:59.952836 5346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:59.952548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:37219 (local address 127.22.37.1:59807)
0620 18:38:59.952663 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:59.952733 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:37219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 18:38:59.953114 5344 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 18:38:59.955026 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 5182
I20260620 18:38:59.955654 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.37.2:41639
--local_ip_for_outbound_sockets=127.22.37.2
--tserver_master_addrs=127.22.37.62:37219
--webserver_port=42065
--webserver_interface=127.22.37.2
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.2.keytab
--principal=kudu/127.22.37.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:36803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:39:00.287252 5347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:00.287623 5347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:00.287797 5347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:00.295257 5347 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:00.295432 5347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:00.295625 5347 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.37.2
I20260620 18:39:00.305627 5347 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:36803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.2.keytab
--principal=kudu/127.22.37.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.37.2:41639
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.37.2
--webserver_port=42065
--tserver_master_addrs=127.22.37.62:37219
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.37.2
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:39:00.307101 5347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:00.308526 5347 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:39:00.322130 5353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:00.322082 5355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:00.322038 5352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:00.323029 5347 server_base.cc:1034] running on GCE node
I20260620 18:39:00.323724 5347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:00.324663 5347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:00.325887 5347 hybrid_clock.cc:648] HybridClock initialized: now 1781980740325824 us; error 59 us; skew 500 ppm
I20260620 18:39:00.326205 5347 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:39:00 dist-test-slave-5lgp krb5kdc[4031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980740, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:39:00.329339 5347 init.cc:377] Logged in from keytab as kudu/127.22.37.2@KRBTEST.COM (short username kudu)
I20260620 18:39:00.330857 5347 webserver.cc:460] Webserver started at http://127.22.37.2:42065/ using document root <none> and password file <none>
I20260620 18:39:00.331553 5347 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:00.331707 5347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:00.337102 5347 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 18:39:00.339964 5362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:00.341296 5347 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:39:00.341543 5347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-1/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-1/wal
uuid: "a50c31a3297f4a65885627657f513f8b"
format_stamp: "Formatted at 2026-06-20 18:38:54 on dist-test-slave-5lgp"
server_key: "10aff9d94f942e7f09b9b7bedb183355"
server_key_iv: "27fe09a0db58516d2f3ed83f473d44b9"
server_key_version: "encryptionkey@0"
I20260620 18:39:00.342540 5347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:00.385887 5347 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:00.390228 5347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:00.390563 5347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:00.391634 5347 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:39:00.392980 5347 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:39:00.393105 5347 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:00.393234 5347 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:39:00.393323 5347 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:00.426824 5347 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.2:41639
I20260620 18:39:00.426875 5474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.2:41639 every 8 connection(s)
I20260620 18:39:00.428321 5347 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-1/data/info.pb
I20260620 18:39:00.429944 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 5347
I20260620 18:39:00.430567 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.37.3:34609
--local_ip_for_outbound_sockets=127.22.37.3
--tserver_master_addrs=127.22.37.62:37219
--webserver_port=43507
--webserver_interface=127.22.37.3
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.3.keytab
--principal=kudu/127.22.37.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:36803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:39:00.437760 5477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:00.435567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:37219 (local address 127.22.37.2:57813)
0620 18:39:00.436449 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:00.436709 (+ 260us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:37219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":361,"threads_started":1}
W20260620 18:39:00.438505 5475 heartbeater.cc:644] Failed to heartbeat to 127.22.37.62:37219 (0 consecutive failures): Network error: Failed to ping master at 127.22.37.62:37219: Client connection negotiation failed: client connection to 127.22.37.62:37219: connect: Connection refused (error 111)
I20260620 18:39:00.439400 5477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:00.439076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:37219 (local address 127.22.37.2:54101)
0620 18:39:00.439212 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:00.439276 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:37219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 18:39:00.440364 5477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:00.439926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:37219 (local address 127.22.37.2:56063)
0620 18:39:00.440102 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:00.440203 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:37219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260620 18:39:00.440614 5475 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 18:39:00.761978 5478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:00.762253 5478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:00.762377 5478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:00.769645 5478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:00.769742 5478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:00.769855 5478 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.37.3
I20260620 18:39:00.779384 5478 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:36803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.3.keytab
--principal=kudu/127.22.37.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.37.3:34609
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.37.3
--webserver_port=43507
--tserver_master_addrs=127.22.37.62:37219
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.37.3
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:39:00.780607 5478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:00.781920 5478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:39:00.795086 5483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:00.795101 5486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:00.795039 5484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:00.796146 5478 server_base.cc:1034] running on GCE node
I20260620 18:39:00.796713 5478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:00.797691 5478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:00.798947 5478 hybrid_clock.cc:648] HybridClock initialized: now 1781980740798887 us; error 62 us; skew 500 ppm
I20260620 18:39:00.799293 5478 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:39:00 dist-test-slave-5lgp krb5kdc[4031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980740, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:39:00.802417 5478 init.cc:377] Logged in from keytab as kudu/127.22.37.3@KRBTEST.COM (short username kudu)
I20260620 18:39:00.803920 5478 webserver.cc:460] Webserver started at http://127.22.37.3:43507/ using document root <none> and password file <none>
I20260620 18:39:00.804613 5478 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:00.804759 5478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:00.810149 5478 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:39:00.812821 5493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:00.814210 5478 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:39:00.814383 5478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-2/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-2/wal
uuid: "113f3aa889864bd291885fb8a53e7f79"
format_stamp: "Formatted at 2026-06-20 18:38:55 on dist-test-slave-5lgp"
server_key: "e9f76a09726ed0c1e0a9d5bdca8d083f"
server_key_iv: "53f9ff7ba04381dfa98bf4cbf99fa2c0"
server_key_version: "encryptionkey@0"
I20260620 18:39:00.815346 5478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:00.832571 5478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:00.836552 5478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:00.836826 5478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:00.837852 5478 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:39:00.839119 5478 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:39:00.839265 5478 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:00.839388 5478 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:39:00.839495 5478 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:00.872287 5478 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.3:34609
I20260620 18:39:00.872342 5605 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.3:34609 every 8 connection(s)
I20260620 18:39:00.873824 5478 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980527431848-22676-0/minicluster-data/ts-2/data/info.pb
I20260620 18:39:00.874477 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 5478
I20260620 18:39:00.878744 5609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:00.876946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:37219 (local address 127.22.37.3:40481)
0620 18:39:00.877706 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:00.877891 (+ 185us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:37219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":341,"threads_started":1}
W20260620 18:39:00.879449 5606 heartbeater.cc:644] Failed to heartbeat to 127.22.37.62:37219 (0 consecutive failures): Network error: Failed to ping master at 127.22.37.62:37219: Client connection negotiation failed: client connection to 127.22.37.62:37219: connect: Connection refused (error 111)
I20260620 18:39:00.880198 5609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:00.879919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:37219 (local address 127.22.37.3:38007)
0620 18:39:00.880041 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:00.880101 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:37219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 18:39:00.881001 5609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:00.880716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:37219 (local address 127.22.37.3:58559)
0620 18:39:00.880821 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:00.880893 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:37219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260620 18:39:00.881273 5606 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 18:39:00.954551 5610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:00.953682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:37219 (local address 127.22.37.1:49143)
0620 18:39:00.954215 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:00.954338 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:37219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":282,"threads_started":1}
I20260620 18:39:01.442193 5611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:01.441297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:37219 (local address 127.22.37.2:36665)
0620 18:39:01.441834 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:01.441964 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:37219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":299,"threads_started":1}
I20260620 18:39:01.882716 5612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:01.881899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:37219 (local address 127.22.37.3:32887)
0620 18:39:01.882325 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:01.882447 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:37219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":217,"threads_started":1}
I20260620 18:39:01.956075 5613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:01.955392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:37219 (local address 127.22.37.1:60153)
0620 18:39:01.955830 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:01.955910 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:37219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":221,"threads_started":1}
I20260620 18:39:02.443921 5614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:02.443232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:37219 (local address 127.22.37.2:40647)
0620 18:39:02.443667 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:02.443755 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:37219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":218,"threads_started":1}
I20260620 18:39:02.884268 5615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:02.883615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:37219 (local address 127.22.37.3:42875)
0620 18:39:02.884035 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:02.884107 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:37219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":206,"threads_started":1}
I20260620 18:39:02.957955 5616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:02.957067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:37219 (local address 127.22.37.1:54343)
0620 18:39:02.957685 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:02.957772 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:37219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":415,"threads_started":1}
I20260620 18:39:03.445699 5617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:03.444914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:37219 (local address 127.22.37.2:47771)
0620 18:39:03.445359 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:03.445524 (+ 165us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:37219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":238,"threads_started":1}
I20260620 18:39:03.885875 5618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:03.885114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:37219 (local address 127.22.37.3:35961)
0620 18:39:03.885622 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:03.885702 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:37219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":312,"threads_started":1}
I20260620 18:39:03.959553 5619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:03.958822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:37219 (local address 127.22.37.1:38915)
0620 18:39:03.959275 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:03.959355 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:37219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":231,"threads_started":1}
I20260620 18:39:04.447794 5620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:04.446861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:37219 (local address 127.22.37.2:43937)
0620 18:39:04.447482 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:04.447567 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:37219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":338,"threads_started":1}
I20260620 18:39:04.887360 5621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:04.886693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:37219 (local address 127.22.37.3:54321)
0620 18:39:04.887105 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:04.887175 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:37219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":205,"threads_started":1}
I20260620 18:39:04.961234 5622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:04.960504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:37219 (local address 127.22.37.1:48969)
0620 18:39:04.960966 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:04.961060 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:37219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":239,"threads_started":1}
I20260620 18:39:05.449496 5623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:05.448732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:37219 (local address 127.22.37.2:45637)
0620 18:39:05.449171 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:05.449250 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:37219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":230,"threads_started":1}
I20260620 18:39:05.889147 5624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:05.888323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:37219 (local address 127.22.37.3:57205)
0620 18:39:05.888896 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:05.888975 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:37219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":350,"threads_started":1}
I20260620 18:39:05.962810 5625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:05.962121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:37219 (local address 127.22.37.1:44353)
0620 18:39:05.962546 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:05.962635 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:37219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":220,"threads_started":1}
I20260620 18:39:06.451093 5626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:06.450365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:37219 (local address 127.22.37.2:43505)
0620 18:39:06.450816 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:06.450911 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:37219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":234,"threads_started":1}
I20260620 18:39:06.890832 5627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:06.889930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:37219 (local address 127.22.37.3:38223)
0620 18:39:06.890590 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:06.890671 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:37219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":442,"threads_started":1}
I20260620 18:39:06.964677 5628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:06.963807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:37219 (local address 127.22.37.1:33737)
0620 18:39:06.964348 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:06.964432 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:37219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":262,"threads_started":1}
I20260620 18:39:07.452874 5629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:07.452050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:37219 (local address 127.22.37.2:35137)
0620 18:39:07.452581 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:07.452660 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:37219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":237,"threads_started":1}
I20260620 18:39:07.892552 5630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:07.891810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:37219 (local address 127.22.37.3:41723)
0620 18:39:07.892256 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:07.892330 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:37219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":196,"threads_started":1}
I20260620 18:39:07.966467 5631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:07.965746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:37219 (local address 127.22.37.1:47535)
0620 18:39:07.966187 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:07.966296 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:37219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":227,"threads_started":1}
I20260620 18:39:08.454497 5632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:08.453795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:37219 (local address 127.22.37.2:59799)
0620 18:39:08.454241 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:08.454327 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:37219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":235,"threads_started":1}
I20260620 18:39:08.894134 5633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:08.893477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:37219 (local address 127.22.37.3:34279)
0620 18:39:08.893894 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:08.893967 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:37219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":221,"threads_started":1}
I20260620 18:39:08.968309 5634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:08.967443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:37219 (local address 127.22.37.1:33365)
0620 18:39:08.968025 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:08.968124 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:37219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":324,"threads_started":1}
I20260620 18:39:09.456188 5635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:09.455425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:37219 (local address 127.22.37.2:52509)
0620 18:39:09.455907 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:09.456012 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:37219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":225,"threads_started":1}
I20260620 18:39:09.895812 5636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:09.894971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:37219 (local address 127.22.37.3:32885)
0620 18:39:09.895547 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:09.895628 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:37219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":318,"threads_started":1}
I20260620 18:39:09.970023 5637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:09.969232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:37219 (local address 127.22.37.1:40547)
0620 18:39:09.969766 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:09.969855 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:37219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":310,"threads_started":1}
I20260620 18:39:10.458017 5638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:10.457283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:37219 (local address 127.22.37.2:56647)
0620 18:39:10.457776 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:10.457856 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:37219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":282,"threads_started":1}
I20260620 18:39:10.897634 5639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:10.896889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:37219 (local address 127.22.37.3:42127)
0620 18:39:10.897323 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:10.897474 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:37219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":238,"threads_started":1}
I20260620 18:39:10.971621 5640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:10.970922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:37219 (local address 127.22.37.1:39483)
0620 18:39:10.971370 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:10.971455 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:37219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":231,"threads_started":1}
I20260620 18:39:11.459916 5641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:11.458945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:37219 (local address 127.22.37.2:47887)
0620 18:39:11.459561 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:11.459691 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:37219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":305,"threads_started":1}
I20260620 18:39:11.899493 5642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:11.898708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:37219 (local address 127.22.37.3:43419)
0620 18:39:11.899206 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:11.899300 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:37219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":247,"threads_started":1}
I20260620 18:39:11.973475 5643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:11.972657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:37219 (local address 127.22.37.1:51271)
0620 18:39:11.973142 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:11.973221 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:37219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":242,"threads_started":1}
I20260620 18:39:12.461711 5644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:12.460910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:37219 (local address 127.22.37.2:44181)
0620 18:39:12.461461 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:12.461551 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:37219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":247,"threads_started":1}
I20260620 18:39:12.901100 5645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:12.900430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:37219 (local address 127.22.37.3:34159)
0620 18:39:12.900854 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:12.900944 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:37219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":222,"threads_started":1}
I20260620 18:39:12.975210 5646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:12.974388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:37219 (local address 127.22.37.1:36365)
0620 18:39:12.974875 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:12.974967 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:37219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":239,"threads_started":1}
I20260620 18:39:13.463482 5647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:13.462664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:37219 (local address 127.22.37.2:48353)
0620 18:39:13.463184 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:13.463294 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:37219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":269,"threads_started":1}
I20260620 18:39:13.902850 5648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:13.902070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:37219 (local address 127.22.37.3:54873)
0620 18:39:13.902565 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:13.902662 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:37219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":257,"threads_started":1}
I20260620 18:39:13.976969 5649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:13.976269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:37219 (local address 127.22.37.1:39329)
0620 18:39:13.976704 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:13.976789 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:37219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":213,"threads_started":1}
I20260620 18:39:14.465299 5650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:14.464529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:37219 (local address 127.22.37.2:55265)
0620 18:39:14.464995 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:14.465084 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:37219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":233,"threads_started":1}
I20260620 18:39:14.904577 5651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:14.903844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:37219 (local address 127.22.37.3:43691)
0620 18:39:14.904321 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:14.904397 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:37219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":238,"threads_started":1}
I20260620 18:39:14.978824 5652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:14.978087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:37219 (local address 127.22.37.1:47247)
0620 18:39:14.978546 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:14.978644 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:37219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":228,"threads_started":1}
I20260620 18:39:15.467283 5653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:15.466464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:37219 (local address 127.22.37.2:37155)
0620 18:39:15.466925 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:15.467025 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:37219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":239,"threads_started":1}
I20260620 18:39:15.880723 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 5182
I20260620 18:39:15.903031 5654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:15.881920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34074 (local address 127.22.37.1:39491)
0620 18:39:15.882541 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:15.882560 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:15.882607 (+ 47us) server_negotiation.cc:407] Connection header received
0620 18:39:15.886043 (+ 3436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:15.886053 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:15.886497 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:15.886796 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:15.888268 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:15.889565 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:15.890786 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:15.891511 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:15.891643 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:15.897271 (+ 5628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:15.897287 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:15.897302 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:15.897354 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:15.900102 (+ 2748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:15.900754 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:15.900760 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:15.900766 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:15.900932 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:15.901254 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:15.901257 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:15.901259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:15.901817 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:15.902049 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:15.902426 (+ 377us) server_negotiation.cc:299] Negotiation successful
0620 18:39:15.902740 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":266,"threads_started":1}
I20260620 18:39:15.904420 5339 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:39:15.906601 5656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:15.905726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:37219 (local address 127.22.37.3:55925)
0620 18:39:15.906226 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:15.906386 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:37219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":257,"threads_started":1}
I20260620 18:39:15.987962 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 5182
I20260620 18:39:16.005185 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 5347
I20260620 18:39:16.022593 5658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:16.006180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60914 (local address 127.22.37.2:41639)
0620 18:39:16.006723 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:16.006738 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:16.006773 (+ 35us) server_negotiation.cc:407] Connection header received
0620 18:39:16.008366 (+ 1593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:16.008375 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:16.008841 (+ 466us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:16.009126 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:16.010582 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:16.011903 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:16.013064 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:16.013749 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:16.013893 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:16.016383 (+ 2490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:16.016389 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:16.016403 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:16.016437 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:16.019560 (+ 3123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:16.020193 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:16.020200 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:16.020205 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:16.020438 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:16.020799 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:16.020802 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:16.020804 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:16.021295 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:16.021590 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:16.021915 (+ 325us) server_negotiation.cc:299] Negotiation successful
0620 18:39:16.022324 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":257,"threads_started":1}
I20260620 18:39:16.024047 5470 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:39:16.107074 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 5347
I20260620 18:39:16.125620 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 5478
I20260620 18:39:16.143637 5660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:16.126671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41674 (local address 127.22.37.3:34609)
0620 18:39:16.127299 (+ 628us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:16.127314 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:16.127357 (+ 43us) server_negotiation.cc:407] Connection header received
0620 18:39:16.128951 (+ 1594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:16.128960 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:16.129517 (+ 557us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:16.129845 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:16.131182 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:16.132418 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:16.133960 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:16.134639 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:16.134790 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:16.137625 (+ 2835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:16.137633 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:16.137648 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:16.137684 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:16.140453 (+ 2769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:16.141135 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:16.141142 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:16.141149 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:16.141322 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:16.141809 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:16.141812 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:16.141814 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:16.142337 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:16.142572 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:16.142894 (+ 322us) server_negotiation.cc:299] Negotiation successful
0620 18:39:16.143347 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":292,"threads_started":1}
I20260620 18:39:16.145073 5601 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:39:16.228405 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 5478
2026-06-20T18:39:16Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24452 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:39:16 dist-test-slave-5lgp krb5kdc[5664](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:39:16 dist-test-slave-5lgp krb5kdc[5664](info): set up 2 sockets
Jun 20 18:39:16 dist-test-slave-5lgp krb5kdc[5664](info): commencing operation
krb5kdc: starting...
W20260620 18:39:17.342126 22676 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/krb5kdc/test-user.keytab.
Jun 20 18:39:17 dist-test-slave-5lgp krb5kdc[5664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980757, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:39:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:39:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.37.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.37.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.37.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab.
Entry for principal kudu/127.22.37.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab.
I20260620 18:39:17.588629 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.62:39147
--webserver_interface=127.22.37.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab
--principal=kudu/127.22.37.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:33907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.37.62:39147
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:39:17.929884 5677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:17.930200 5677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:17.930406 5677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:17.937834 5677 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 18:39:17.937944 5677 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 18:39:17.938023 5677 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 18:39:17.938097 5677 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:39:17.938216 5677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:17.938326 5677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:17.938421 5677 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:39:17.938501 5677 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:39:17.948611 5677 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:33907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.22.37.62:39147
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab
--principal=kudu/127.22.37.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.62:39147
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.37.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:39:17.950018 5677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:17.951445 5677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:39:17.964406 5682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:17.964264 5683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:17.964566 5685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:17.965269 5677 server_base.cc:1034] running on GCE node
I20260620 18:39:17.966235 5677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:17.967731 5677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:17.968954 5677 hybrid_clock.cc:648] HybridClock initialized: now 1781980757968892 us; error 31 us; skew 500 ppm
I20260620 18:39:17.969311 5677 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:39:17 dist-test-slave-5lgp krb5kdc[5664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980757, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:39:17.972429 5677 init.cc:377] Logged in from keytab as kudu/127.22.37.62@KRBTEST.COM (short username kudu)
I20260620 18:39:17.973853 5677 webserver.cc:460] Webserver started at http://127.22.37.62:34685/ using document root <none> and password file <none>
I20260620 18:39:17.974500 5677 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:17.974584 5677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:17.974849 5677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:39:17.977910 5677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/master-0/data/instance:
uuid: "a593d764462a463d8334200095758268"
format_stamp: "Formatted at 2026-06-20 18:39:17 on dist-test-slave-5lgp"
server_key: "9262f4c61f5b39011a2d4c9108445344"
server_key_iv: "f62851751da673374e04ecb83bbba083"
server_key_version: "encryptionkey@0"
I20260620 18:39:17.978713 5677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/master-0/wal/instance:
uuid: "a593d764462a463d8334200095758268"
format_stamp: "Formatted at 2026-06-20 18:39:17 on dist-test-slave-5lgp"
server_key: "9262f4c61f5b39011a2d4c9108445344"
server_key_iv: "f62851751da673374e04ecb83bbba083"
server_key_version: "encryptionkey@0"
I20260620 18:39:17.984036 5677 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260620 18:39:17.987473 5692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:17.988883 5677 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:39:17.989034 5677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/master-0/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/master-0/wal
uuid: "a593d764462a463d8334200095758268"
format_stamp: "Formatted at 2026-06-20 18:39:17 on dist-test-slave-5lgp"
server_key: "9262f4c61f5b39011a2d4c9108445344"
server_key_iv: "f62851751da673374e04ecb83bbba083"
server_key_version: "encryptionkey@0"
I20260620 18:39:17.989230 5677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:18.014214 5677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:18.018594 5677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:18.018875 5677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:18.038903 5677 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.62:39147
I20260620 18:39:18.038926 5743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.62:39147 every 8 connection(s)
I20260620 18:39:18.040493 5677 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
I20260620 18:39:18.042560 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 5677
I20260620 18:39:18.042805 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/master-0/wal/instance
I20260620 18:39:18.043484 22676 external_mini_cluster.cc:1428] Setting key b848deec3571132b300766bb226e796e
I20260620 18:39:18.045099 5744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 18:39:18 dist-test-slave-5lgp krb5kdc[5664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980757, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM
I20260620 18:39:18.060613 5744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a593d764462a463d8334200095758268: Bootstrap starting.
I20260620 18:39:18.064128 5744 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a593d764462a463d8334200095758268: Neither blocks nor log segments found. Creating new log.
I20260620 18:39:18.065634 5744 log.cc:824] T 00000000000000000000000000000000 P a593d764462a463d8334200095758268: Log is configured to *not* fsync() on all Append() calls
I20260620 18:39:18.069134 5744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a593d764462a463d8334200095758268: No bootstrap required, opened a new log
I20260620 18:39:18.069185 5747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:18.045655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38258 (local address 127.22.37.62:39147)
0620 18:39:18.046477 (+ 822us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:18.046489 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:18.046533 (+ 44us) server_negotiation.cc:407] Connection header received
0620 18:39:18.048218 (+ 1685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:18.048244 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:18.048711 (+ 467us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:18.049093 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:18.051161 (+ 2068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:18.052596 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:18.054628 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:18.055383 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:18.055532 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:18.060955 (+ 5423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:18.060965 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:18.060983 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:18.061029 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:18.064312 (+ 3283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:18.065213 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:18.065221 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:18.065232 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:18.065491 (+ 259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:18.065994 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:18.065998 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:18.066001 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:18.066891 (+ 890us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:18.067255 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:18.067670 (+ 415us) server_negotiation.cc:299] Negotiation successful
0620 18:39:18.068110 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":366,"threads_started":1}
I20260620 18:39:18.077754 5744 raft_consensus.cc:348] T 00000000000000000000000000000000 P a593d764462a463d8334200095758268 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a593d764462a463d8334200095758268" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 39147 } }
I20260620 18:39:18.078161 5744 raft_consensus.cc:374] T 00000000000000000000000000000000 P a593d764462a463d8334200095758268 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:39:18.078287 5744 raft_consensus.cc:729] T 00000000000000000000000000000000 P a593d764462a463d8334200095758268 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a593d764462a463d8334200095758268, State: Initialized, Role: FOLLOWER
I20260620 18:39:18.078878 5744 consensus_queue.cc:260] T 00000000000000000000000000000000 P a593d764462a463d8334200095758268 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a593d764462a463d8334200095758268" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 39147 } }
I20260620 18:39:18.079113 5744 raft_consensus.cc:388] T 00000000000000000000000000000000 P a593d764462a463d8334200095758268 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:39:18.079259 5744 raft_consensus.cc:482] T 00000000000000000000000000000000 P a593d764462a463d8334200095758268 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:39:18.079442 5744 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a593d764462a463d8334200095758268 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:39:18.082116 5744 raft_consensus.cc:504] T 00000000000000000000000000000000 P a593d764462a463d8334200095758268 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a593d764462a463d8334200095758268" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 39147 } }
I20260620 18:39:18.082638 5744 leader_election.cc:302] T 00000000000000000000000000000000 P a593d764462a463d8334200095758268 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a593d764462a463d8334200095758268; no voters:
I20260620 18:39:18.083288 5744 leader_election.cc:288] T 00000000000000000000000000000000 P a593d764462a463d8334200095758268 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:39:18.083482 5749 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a593d764462a463d8334200095758268 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:39:18.084167 5749 raft_consensus.cc:686] T 00000000000000000000000000000000 P a593d764462a463d8334200095758268 [term 1 LEADER]: Becoming Leader. State: Replica: a593d764462a463d8334200095758268, State: Running, Role: LEADER
I20260620 18:39:18.084738 5749 consensus_queue.cc:237] T 00000000000000000000000000000000 P a593d764462a463d8334200095758268 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a593d764462a463d8334200095758268" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 39147 } }
I20260620 18:39:18.085443 5744 sys_catalog.cc:564] T 00000000000000000000000000000000 P a593d764462a463d8334200095758268 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:39:18.088506 5751 sys_catalog.cc:455] T 00000000000000000000000000000000 P a593d764462a463d8334200095758268 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a593d764462a463d8334200095758268. Latest consensus state: current_term: 1 leader_uuid: "a593d764462a463d8334200095758268" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a593d764462a463d8334200095758268" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 39147 } } }
I20260620 18:39:18.088577 5750 sys_catalog.cc:455] T 00000000000000000000000000000000 P a593d764462a463d8334200095758268 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a593d764462a463d8334200095758268" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a593d764462a463d8334200095758268" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 39147 } } }
I20260620 18:39:18.089059 5751 sys_catalog.cc:458] T 00000000000000000000000000000000 P a593d764462a463d8334200095758268 [sys.catalog]: This master's current role is: LEADER
I20260620 18:39:18.089067 5750 sys_catalog.cc:458] T 00000000000000000000000000000000 P a593d764462a463d8334200095758268 [sys.catalog]: This master's current role is: LEADER
I20260620 18:39:18.089986 5758 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:39:18.095858 5758 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:39:18.104898 5758 catalog_manager.cc:1347] Generated new cluster ID: a5cfbde2c9ad4e62b179da5e8af50d12
I20260620 18:39:18.105010 5758 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:39:18.119747 5758 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:39:18.121694 5758 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:39:18.138314 5758 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a593d764462a463d8334200095758268: Generated new TSK 0
I20260620 18:39:18.139432 5758 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.37.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.37.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.37.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.1.keytab.
Entry for principal kudu/127.22.37.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.1.keytab.
I20260620 18:39:18.206740 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.37.1:0
--local_ip_for_outbound_sockets=127.22.37.1
--webserver_interface=127.22.37.1
--webserver_port=0
--tserver_master_addrs=127.22.37.62:39147
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.1.keytab
--principal=kudu/127.22.37.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:33907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:39:18.537088 5770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:18.537446 5770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:18.537616 5770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:18.544786 5770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:18.544940 5770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:18.545111 5770 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.37.1
I20260620 18:39:18.556180 5770 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:33907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.1.keytab
--principal=kudu/127.22.37.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.37.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.37.1
--webserver_port=0
--tserver_master_addrs=127.22.37.62:39147
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.37.1
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:39:18.557590 5770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:18.559090 5770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:39:18.572638 5778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:18.572561 5776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:18.572582 5775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:18.573678 5770 server_base.cc:1034] running on GCE node
I20260620 18:39:18.574319 5770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:18.575255 5770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:18.576514 5770 hybrid_clock.cc:648] HybridClock initialized: now 1781980758576449 us; error 50 us; skew 500 ppm
I20260620 18:39:18.576869 5770 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:39:18 dist-test-slave-5lgp krb5kdc[5664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980758, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:39:18.579960 5770 init.cc:377] Logged in from keytab as kudu/127.22.37.1@KRBTEST.COM (short username kudu)
I20260620 18:39:18.581496 5770 webserver.cc:460] Webserver started at http://127.22.37.1:46361/ using document root <none> and password file <none>
I20260620 18:39:18.582158 5770 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:18.582311 5770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:18.582625 5770 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:39:18.585654 5770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-0/data/instance:
uuid: "129d88bd84a241a68291203014b805d0"
format_stamp: "Formatted at 2026-06-20 18:39:18 on dist-test-slave-5lgp"
server_key: "5fa8a0c42fb597c961965ef437c4e92a"
server_key_iv: "a199ce8a6a32e88fd610debef841e3df"
server_key_version: "encryptionkey@0"
I20260620 18:39:18.586407 5770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-0/wal/instance:
uuid: "129d88bd84a241a68291203014b805d0"
format_stamp: "Formatted at 2026-06-20 18:39:18 on dist-test-slave-5lgp"
server_key: "5fa8a0c42fb597c961965ef437c4e92a"
server_key_iv: "a199ce8a6a32e88fd610debef841e3df"
server_key_version: "encryptionkey@0"
I20260620 18:39:18.591518 5770 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260620 18:39:18.595036 5785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:18.596303 5770 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 18:39:18.596468 5770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-0/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-0/wal
uuid: "129d88bd84a241a68291203014b805d0"
format_stamp: "Formatted at 2026-06-20 18:39:18 on dist-test-slave-5lgp"
server_key: "5fa8a0c42fb597c961965ef437c4e92a"
server_key_iv: "a199ce8a6a32e88fd610debef841e3df"
server_key_version: "encryptionkey@0"
I20260620 18:39:18.596675 5770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:18.616487 5770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:18.620781 5770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:18.621105 5770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:18.622107 5770 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:39:18.623454 5770 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:39:18.623556 5770 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:18.623708 5770 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:39:18.623777 5770 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:18.657843 5770 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.1:42741
I20260620 18:39:18.657892 5897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.1:42741 every 8 connection(s)
I20260620 18:39:18.659150 5770 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-0/data/info.pb
I20260620 18:39:18.662438 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 5770
I20260620 18:39:18.662668 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-0/wal/instance
I20260620 18:39:18.663339 22676 external_mini_cluster.cc:1428] Setting key 75828aee059fbde34bbc74de1deec300
Jun 20 18:39:18 dist-test-slave-5lgp krb5kdc[5664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980758, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.1@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM
I20260620 18:39:18.679606 5900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:18.661877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.1:35183 (local address 127.22.37.62:39147)
0620 18:39:18.662410 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:18.662419 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:18.663888 (+ 1469us) server_negotiation.cc:407] Connection header received
0620 18:39:18.665676 (+ 1788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:18.665682 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:18.665794 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:18.665933 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:18.667422 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:18.668109 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:18.669656 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:18.670035 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:18.670166 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:18.673805 (+ 3639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:18.673810 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:18.673813 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:18.673844 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:18.676331 (+ 2487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:18.677260 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:18.677264 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:18.677266 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:18.677437 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:18.677887 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:18.677890 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:18.677892 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:18.678145 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:18.678276 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:18.679164 (+ 888us) server_negotiation.cc:299] Negotiation successful
0620 18:39:18.679341 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":240,"threads_started":1}
I20260620 18:39:18.681043 5901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:18.662457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:39147 (local address 127.22.37.1:35183)
0620 18:39:18.663319 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:18.663371 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:18.665357 (+ 1986us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:18.666182 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:18.666191 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:18.666689 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:18.667232 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:18.667253 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:18.668265 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:18.668270 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:18.669520 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:18.669532 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:18.670252 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:18.670262 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:18.670512 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:18.671159 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:18.671186 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:18.673592 (+ 2406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:39:18.676467 (+ 2875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:18.676478 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:18.676496 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:18.677125 (+ 629us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:18.677540 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:18.677547 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:18.677553 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:18.677777 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:18.678300 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:39:18.678310 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:39:18.678750 (+ 440us) client_negotiation.cc:769] Sending connection context
0620 18:39:18.679032 (+ 282us) client_negotiation.cc:240] Negotiation successful
0620 18:39:18.679586 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":365,"threads_started":1}
I20260620 18:39:18.682623 5898 heartbeater.cc:347] Connected to a master server at 127.22.37.62:39147
I20260620 18:39:18.682924 5898 heartbeater.cc:464] Registering TS with master...
I20260620 18:39:18.683637 5898 heartbeater.cc:511] Master 127.22.37.62:39147 requested a full tablet report, sending...
I20260620 18:39:18.685958 5709 ts_manager.cc:194] Registered new tserver with Master: 129d88bd84a241a68291203014b805d0 (127.22.37.1:42741)
I20260620 18:39:18.687513 5709 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.37.1@KRBTEST.COM'} at 127.22.37.1:35183
WARNING: no policy specified for kudu/127.22.37.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.37.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.37.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.2.keytab.
Entry for principal kudu/127.22.37.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.2.keytab.
I20260620 18:39:18.716858 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.37.2:0
--local_ip_for_outbound_sockets=127.22.37.2
--webserver_interface=127.22.37.2
--webserver_port=0
--tserver_master_addrs=127.22.37.62:39147
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.2.keytab
--principal=kudu/127.22.37.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:33907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:39:19.034906 5904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:19.035234 5904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:19.035406 5904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:19.042784 5904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:19.042933 5904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:19.043119 5904 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.37.2
I20260620 18:39:19.052879 5904 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:33907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.2.keytab
--principal=kudu/127.22.37.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.37.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.37.2
--webserver_port=0
--tserver_master_addrs=127.22.37.62:39147
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.37.2
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:39:19.054281 5904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:19.055620 5904 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:39:19.069267 5909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:19.069175 5910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:19.069238 5912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:19.070070 5904 server_base.cc:1034] running on GCE node
I20260620 18:39:19.070744 5904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:19.071707 5904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:19.072965 5904 hybrid_clock.cc:648] HybridClock initialized: now 1781980759072900 us; error 57 us; skew 500 ppm
I20260620 18:39:19.073314 5904 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:39:19 dist-test-slave-5lgp krb5kdc[5664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980759, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:39:19.076526 5904 init.cc:377] Logged in from keytab as kudu/127.22.37.2@KRBTEST.COM (short username kudu)
I20260620 18:39:19.079134 5904 webserver.cc:460] Webserver started at http://127.22.37.2:44147/ using document root <none> and password file <none>
I20260620 18:39:19.080128 5904 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:19.080384 5904 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:19.080811 5904 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:39:19.085709 5904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-1/data/instance:
uuid: "c995f8a8741a4540b650832e93ff4d36"
format_stamp: "Formatted at 2026-06-20 18:39:19 on dist-test-slave-5lgp"
server_key: "1125f2023af9f5c615e75957e6bcb2aa"
server_key_iv: "e593df8bfc77c25c10e5285125c17478"
server_key_version: "encryptionkey@0"
I20260620 18:39:19.086913 5904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-1/wal/instance:
uuid: "c995f8a8741a4540b650832e93ff4d36"
format_stamp: "Formatted at 2026-06-20 18:39:19 on dist-test-slave-5lgp"
server_key: "1125f2023af9f5c615e75957e6bcb2aa"
server_key_iv: "e593df8bfc77c25c10e5285125c17478"
server_key_version: "encryptionkey@0"
I20260620 18:39:19.095152 5904 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.004s sys 0.004s
I20260620 18:39:19.099992 5919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:19.101810 5904 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260620 18:39:19.101972 5904 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-1/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-1/wal
uuid: "c995f8a8741a4540b650832e93ff4d36"
format_stamp: "Formatted at 2026-06-20 18:39:19 on dist-test-slave-5lgp"
server_key: "1125f2023af9f5c615e75957e6bcb2aa"
server_key_iv: "e593df8bfc77c25c10e5285125c17478"
server_key_version: "encryptionkey@0"
I20260620 18:39:19.102265 5904 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:19.122365 5904 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:19.126523 5904 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:19.126816 5904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:19.127887 5904 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:39:19.129168 5904 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:39:19.129247 5904 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:19.129433 5904 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:39:19.129487 5904 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260620 18:39:19.168107 5904 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.2:36829
I20260620 18:39:19.168162 6031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.2:36829 every 8 connection(s)
I20260620 18:39:19.169795 5904 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-1/data/info.pb
I20260620 18:39:19.179150 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 5904
I20260620 18:39:19.179348 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-1/wal/instance
I20260620 18:39:19.179809 22676 external_mini_cluster.cc:1428] Setting key 3b0fd82810d3dfec3fcd737dcc969880
Jun 20 18:39:19 dist-test-slave-5lgp krb5kdc[5664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980759, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.2@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM
I20260620 18:39:19.193248 5900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:19.172878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.2:36375 (local address 127.22.37.62:39147)
0620 18:39:19.173072 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:19.173078 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:19.174661 (+ 1583us) server_negotiation.cc:407] Connection header received
0620 18:39:19.177112 (+ 2451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:19.177117 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:19.177208 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:19.177353 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:19.178853 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:19.179703 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:19.181469 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:19.181964 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:19.182112 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:19.186309 (+ 4197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:19.186318 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:19.186326 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:19.186369 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:19.189942 (+ 3573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:19.190840 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:19.190843 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:19.190846 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:19.190972 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:19.191494 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:19.191497 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:19.191499 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:19.191758 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:19.191869 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:19.192829 (+ 960us) server_negotiation.cc:299] Negotiation successful
0620 18:39:19.193012 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 18:39:19.194772 6034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:19.173507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:39147 (local address 127.22.37.2:36375)
0620 18:39:19.174464 (+ 957us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:19.174530 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:19.176812 (+ 2282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:19.177616 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:19.177624 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:19.178109 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:19.178671 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:19.178688 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:19.179851 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:19.179856 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:19.181277 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:19.181289 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:19.182206 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:19.182213 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:19.182526 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:19.183220 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:19.183254 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:19.186074 (+ 2820us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:39:19.190116 (+ 4042us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:19.190123 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:19.190135 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:19.190708 (+ 573us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:19.191076 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:19.191083 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:19.191088 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:19.191359 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:19.191898 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:39:19.191908 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:39:19.192378 (+ 470us) client_negotiation.cc:769] Sending connection context
0620 18:39:19.192677 (+ 299us) client_negotiation.cc:240] Negotiation successful
0620 18:39:19.193255 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":625,"thread_start_us":379,"threads_started":1}
I20260620 18:39:19.196383 6032 heartbeater.cc:347] Connected to a master server at 127.22.37.62:39147
I20260620 18:39:19.196729 6032 heartbeater.cc:464] Registering TS with master...
I20260620 18:39:19.197559 6032 heartbeater.cc:511] Master 127.22.37.62:39147 requested a full tablet report, sending...
I20260620 18:39:19.199160 5709 ts_manager.cc:194] Registered new tserver with Master: c995f8a8741a4540b650832e93ff4d36 (127.22.37.2:36829)
I20260620 18:39:19.200099 5709 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.37.2@KRBTEST.COM'} at 127.22.37.2:36375
WARNING: no policy specified for kudu/127.22.37.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.37.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.37.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.3.keytab.
Entry for principal kudu/127.22.37.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.3.keytab.
I20260620 18:39:19.234462 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.37.3:0
--local_ip_for_outbound_sockets=127.22.37.3
--webserver_interface=127.22.37.3
--webserver_port=0
--tserver_master_addrs=127.22.37.62:39147
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.3.keytab
--principal=kudu/127.22.37.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:33907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:39:19.558923 6037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:19.559266 6037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:19.559445 6037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:19.567039 6037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:19.567214 6037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:19.567391 6037 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.37.3
I20260620 18:39:19.577576 6037 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:33907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.3.keytab
--principal=kudu/127.22.37.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.37.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.37.3
--webserver_port=0
--tserver_master_addrs=127.22.37.62:39147
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.37.3
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:39:19.578969 6037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:19.580363 6037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:39:19.594000 6042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:19.593890 6045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:19.593909 6043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:19.595079 6037 server_base.cc:1034] running on GCE node
I20260620 18:39:19.595745 6037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:19.596696 6037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:19.597949 6037 hybrid_clock.cc:648] HybridClock initialized: now 1781980759597877 us; error 62 us; skew 500 ppm
I20260620 18:39:19.598299 6037 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:39:19 dist-test-slave-5lgp krb5kdc[5664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980759, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:39:19.601701 6037 init.cc:377] Logged in from keytab as kudu/127.22.37.3@KRBTEST.COM (short username kudu)
I20260620 18:39:19.603226 6037 webserver.cc:460] Webserver started at http://127.22.37.3:36447/ using document root <none> and password file <none>
I20260620 18:39:19.603943 6037 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:19.604089 6037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:19.604409 6037 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:39:19.607493 6037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-2/data/instance:
uuid: "51e1565c5d3a45da9d90ee7d50e5184e"
format_stamp: "Formatted at 2026-06-20 18:39:19 on dist-test-slave-5lgp"
server_key: "27ff70d06d7fdb110dbc6b59aeaf247c"
server_key_iv: "3b466baf5f24c2a8f59829527b5af6de"
server_key_version: "encryptionkey@0"
I20260620 18:39:19.608330 6037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-2/wal/instance:
uuid: "51e1565c5d3a45da9d90ee7d50e5184e"
format_stamp: "Formatted at 2026-06-20 18:39:19 on dist-test-slave-5lgp"
server_key: "27ff70d06d7fdb110dbc6b59aeaf247c"
server_key_iv: "3b466baf5f24c2a8f59829527b5af6de"
server_key_version: "encryptionkey@0"
I20260620 18:39:19.613636 6037 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260620 18:39:19.617146 6052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:19.618678 6037 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:39:19.618860 6037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-2/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-2/wal
uuid: "51e1565c5d3a45da9d90ee7d50e5184e"
format_stamp: "Formatted at 2026-06-20 18:39:19 on dist-test-slave-5lgp"
server_key: "27ff70d06d7fdb110dbc6b59aeaf247c"
server_key_iv: "3b466baf5f24c2a8f59829527b5af6de"
server_key_version: "encryptionkey@0"
I20260620 18:39:19.619060 6037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:19.640988 6037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:19.645236 6037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:19.645608 6037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:19.646636 6037 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:39:19.647972 6037 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:39:19.648077 6037 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:19.648229 6037 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:39:19.648345 6037 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:19.683480 6037 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.3:35831
I20260620 18:39:19.683573 6164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.3:35831 every 8 connection(s)
I20260620 18:39:19.684814 6037 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-2/data/info.pb
I20260620 18:39:19.686632 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 6037
I20260620 18:39:19.686833 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-2/wal/instance
I20260620 18:39:19.687326 22676 external_mini_cluster.cc:1428] Setting key 0dd55afa4755f13b2796417384850e56
I20260620 18:39:19.691022 5898 heartbeater.cc:503] Master 127.22.37.62:39147 was elected leader, sending a full tablet report...
Jun 20 18:39:19 dist-test-slave-5lgp krb5kdc[5664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980759, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.3@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM
I20260620 18:39:19.708385 5900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:19.689330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.3:45799 (local address 127.22.37.62:39147)
0620 18:39:19.689562 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:19.689567 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:19.691192 (+ 1625us) server_negotiation.cc:407] Connection header received
0620 18:39:19.693709 (+ 2517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:19.693716 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:19.693802 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:19.693896 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:19.695450 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:19.696210 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:19.697836 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:19.698280 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:19.698417 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:19.702603 (+ 4186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:19.702609 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:19.702616 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:19.702664 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:19.705258 (+ 2594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:19.706044 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:19.706049 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:19.706051 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:19.706172 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:19.706592 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:19.706595 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:19.706597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:19.706860 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:19.706974 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:19.707818 (+ 844us) server_negotiation.cc:299] Negotiation successful
0620 18:39:19.708013 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260620 18:39:19.709287 6167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:19.690128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:39147 (local address 127.22.37.3:45799)
0620 18:39:19.691009 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:19.691063 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:19.693324 (+ 2261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:19.694158 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:19.694167 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:19.694704 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:19.695285 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:19.695303 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:19.696369 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:19.696374 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:19.697693 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:19.697705 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:19.698507 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:19.698515 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:19.698865 (+ 350us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:19.699752 (+ 887us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:19.699785 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:19.702421 (+ 2636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:39:19.705451 (+ 3030us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:19.705458 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:19.705472 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:19.705920 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:19.706294 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:19.706298 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:19.706300 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:19.706487 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:19.706989 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:39:19.706995 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:39:19.707407 (+ 412us) client_negotiation.cc:769] Sending connection context
0620 18:39:19.707651 (+ 244us) client_negotiation.cc:240] Negotiation successful
0620 18:39:19.708072 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":316,"threads_started":1}
I20260620 18:39:19.710872 6165 heartbeater.cc:347] Connected to a master server at 127.22.37.62:39147
I20260620 18:39:19.711256 6165 heartbeater.cc:464] Registering TS with master...
I20260620 18:39:19.712008 6165 heartbeater.cc:511] Master 127.22.37.62:39147 requested a full tablet report, sending...
I20260620 18:39:19.713757 5709 ts_manager.cc:194] Registered new tserver with Master: 51e1565c5d3a45da9d90ee7d50e5184e (127.22.37.3:35831)
I20260620 18:39:19.714926 5709 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.37.3@KRBTEST.COM'} at 127.22.37.3:45799
I20260620 18:39:19.716234 22676 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:39:19.735376 5900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:19.720980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57896 (local address 127.22.37.62:39147)
0620 18:39:19.721221 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:19.721231 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:19.721681 (+ 450us) server_negotiation.cc:407] Connection header received
0620 18:39:19.721888 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:19.721892 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:19.721956 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:19.722035 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:19.723787 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:19.724545 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:19.726083 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:19.726507 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:19.726670 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:19.729619 (+ 2949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:19.729625 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:19.729629 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:19.729670 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:19.732290 (+ 2620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:19.733077 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:19.733083 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:19.733086 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:19.733228 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:19.733714 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:19.733725 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:19.733727 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:19.734080 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:19.734362 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:19.734748 (+ 386us) server_negotiation.cc:299] Negotiation successful
0620 18:39:19.734981 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 18:39:19.739785 5709 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57896:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:39:19.747364 5709 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:39:19.782357 6180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:19.759600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.1:42741 (local address 127.0.0.1:52132)
0620 18:39:19.774306 (+ 14706us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:19.774324 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:19.774438 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:19.775613 (+ 1175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:19.775618 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:19.775644 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:39:19.776145 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:19.776159 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:19.778060 (+ 1901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:19.778068 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:19.779833 (+ 1765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:19.779848 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:19.781209 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:19.781215 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:19.781431 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:19.781496 (+ 65us) client_negotiation.cc:769] Sending connection context
0620 18:39:19.781646 (+ 150us) client_negotiation.cc:240] Negotiation successful
0620 18:39:19.781826 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14576,"mutex_wait_us":76,"thread_start_us":10735,"threads_started":1}
I20260620 18:39:19.783097 6178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:19.759476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.3:35831 (local address 127.0.0.1:36178)
0620 18:39:19.770760 (+ 11284us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:19.770797 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:19.770978 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:19.774492 (+ 3514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:19.774503 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:19.774543 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:39:19.775038 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:19.775058 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:19.777352 (+ 2294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:19.777359 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:19.779983 (+ 2624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:19.779996 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:19.782235 (+ 2239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:19.782241 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:19.782435 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:19.782466 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 18:39:19.782547 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 18:39:19.782712 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10998,"mutex_wait_us":3,"thread_start_us":16631,"threads_started":1}
I20260620 18:39:19.783324 6181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:19.760328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52132 (local address 127.22.37.1:42741)
0620 18:39:19.775103 (+ 14775us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:19.775111 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:19.775139 (+ 28us) server_negotiation.cc:407] Connection header received
0620 18:39:19.775238 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:19.775244 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:19.775430 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:19.775634 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:39:19.776394 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:19.777909 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:19.780000 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:19.780987 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:19.781141 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:19.781413 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:19.781708 (+ 295us) server_negotiation.cc:299] Negotiation successful
0620 18:39:19.782971 (+ 1263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14602,"thread_start_us":14369,"threads_started":1}
I20260620 18:39:19.783847 6179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:19.759673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36178 (local address 127.22.37.3:35831)
0620 18:39:19.773261 (+ 13588us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:19.773274 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:19.773317 (+ 43us) server_negotiation.cc:407] Connection header received
0620 18:39:19.773463 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:19.773475 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:19.773842 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:19.774078 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:39:19.775299 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:19.777149 (+ 1850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:19.780191 (+ 3042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:19.781685 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:19.781834 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:19.782065 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:19.783343 (+ 1278us) server_negotiation.cc:299] Negotiation successful
0620 18:39:19.783514 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13346,"thread_start_us":13154,"threads_started":1}
I20260620 18:39:19.784714 6177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:19.760039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.2:36829 (local address 127.0.0.1:41496)
0620 18:39:19.776400 (+ 16361us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:19.776413 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:19.776535 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:19.777024 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:19.777029 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:19.777063 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:39:19.777531 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:19.777545 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:19.779485 (+ 1940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:19.779491 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:19.781730 (+ 2239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:19.781746 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:19.783921 (+ 2175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:19.783927 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:19.784154 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:19.784176 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 18:39:19.784276 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 18:39:19.784395 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16231,"thread_start_us":13753,"threads_started":1}
I20260620 18:39:19.784787 6182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:19.760652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41496 (local address 127.22.37.2:36829)
0620 18:39:19.773584 (+ 12932us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:19.773596 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:19.776512 (+ 2916us) server_negotiation.cc:407] Connection header received
0620 18:39:19.776684 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:19.776689 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:19.776858 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:19.777037 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:39:19.777770 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:19.779294 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:19.781906 (+ 2612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:19.783452 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:19.783551 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:19.783742 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:19.784336 (+ 594us) server_negotiation.cc:299] Negotiation successful
0620 18:39:19.784503 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12738,"thread_start_us":12596,"threads_started":1}
I20260620 18:39:19.787427 5833 tablet_service.cc:1467] Processing CreateTablet for tablet c99c11c996b140d3aaa884c41fabe2e3 (DEFAULT_TABLE table=test-table [id=7b23d07f173346e6811c590def77d78c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:39:19.788197 6100 tablet_service.cc:1467] Processing CreateTablet for tablet c99c11c996b140d3aaa884c41fabe2e3 (DEFAULT_TABLE table=test-table [id=7b23d07f173346e6811c590def77d78c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:39:19.789024 5833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c99c11c996b140d3aaa884c41fabe2e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:39:19.788594 5967 tablet_service.cc:1467] Processing CreateTablet for tablet c99c11c996b140d3aaa884c41fabe2e3 (DEFAULT_TABLE table=test-table [id=7b23d07f173346e6811c590def77d78c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:39:19.789618 6100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c99c11c996b140d3aaa884c41fabe2e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:39:19.789916 5967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c99c11c996b140d3aaa884c41fabe2e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:39:19.801203 6183 tablet_bootstrap.cc:492] T c99c11c996b140d3aaa884c41fabe2e3 P 51e1565c5d3a45da9d90ee7d50e5184e: Bootstrap starting.
I20260620 18:39:19.804311 6184 tablet_bootstrap.cc:492] T c99c11c996b140d3aaa884c41fabe2e3 P 129d88bd84a241a68291203014b805d0: Bootstrap starting.
I20260620 18:39:19.805085 6185 tablet_bootstrap.cc:492] T c99c11c996b140d3aaa884c41fabe2e3 P c995f8a8741a4540b650832e93ff4d36: Bootstrap starting.
I20260620 18:39:19.805609 6183 tablet_bootstrap.cc:654] T c99c11c996b140d3aaa884c41fabe2e3 P 51e1565c5d3a45da9d90ee7d50e5184e: Neither blocks nor log segments found. Creating new log.
I20260620 18:39:19.806993 6183 log.cc:824] T c99c11c996b140d3aaa884c41fabe2e3 P 51e1565c5d3a45da9d90ee7d50e5184e: Log is configured to *not* fsync() on all Append() calls
I20260620 18:39:19.808483 6184 tablet_bootstrap.cc:654] T c99c11c996b140d3aaa884c41fabe2e3 P 129d88bd84a241a68291203014b805d0: Neither blocks nor log segments found. Creating new log.
I20260620 18:39:19.808990 6185 tablet_bootstrap.cc:654] T c99c11c996b140d3aaa884c41fabe2e3 P c995f8a8741a4540b650832e93ff4d36: Neither blocks nor log segments found. Creating new log.
I20260620 18:39:19.809666 6184 log.cc:824] T c99c11c996b140d3aaa884c41fabe2e3 P 129d88bd84a241a68291203014b805d0: Log is configured to *not* fsync() on all Append() calls
I20260620 18:39:19.810333 6185 log.cc:824] T c99c11c996b140d3aaa884c41fabe2e3 P c995f8a8741a4540b650832e93ff4d36: Log is configured to *not* fsync() on all Append() calls
I20260620 18:39:19.812493 6183 tablet_bootstrap.cc:492] T c99c11c996b140d3aaa884c41fabe2e3 P 51e1565c5d3a45da9d90ee7d50e5184e: No bootstrap required, opened a new log
I20260620 18:39:19.812857 6183 ts_tablet_manager.cc:1397] T c99c11c996b140d3aaa884c41fabe2e3 P 51e1565c5d3a45da9d90ee7d50e5184e: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.006s
I20260620 18:39:19.816001 6185 tablet_bootstrap.cc:492] T c99c11c996b140d3aaa884c41fabe2e3 P c995f8a8741a4540b650832e93ff4d36: No bootstrap required, opened a new log
I20260620 18:39:19.816001 6184 tablet_bootstrap.cc:492] T c99c11c996b140d3aaa884c41fabe2e3 P 129d88bd84a241a68291203014b805d0: No bootstrap required, opened a new log
I20260620 18:39:19.816285 6184 ts_tablet_manager.cc:1397] T c99c11c996b140d3aaa884c41fabe2e3 P 129d88bd84a241a68291203014b805d0: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.006s
I20260620 18:39:19.816283 6185 ts_tablet_manager.cc:1397] T c99c11c996b140d3aaa884c41fabe2e3 P c995f8a8741a4540b650832e93ff4d36: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.006s
I20260620 18:39:19.821086 6183 raft_consensus.cc:348] T c99c11c996b140d3aaa884c41fabe2e3 P 51e1565c5d3a45da9d90ee7d50e5184e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51e1565c5d3a45da9d90ee7d50e5184e" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35831 } } peers { permanent_uuid: "129d88bd84a241a68291203014b805d0" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 42741 } } peers { permanent_uuid: "c995f8a8741a4540b650832e93ff4d36" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 36829 } }
I20260620 18:39:19.821472 6183 raft_consensus.cc:374] T c99c11c996b140d3aaa884c41fabe2e3 P 51e1565c5d3a45da9d90ee7d50e5184e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:39:19.821574 6183 raft_consensus.cc:729] T c99c11c996b140d3aaa884c41fabe2e3 P 51e1565c5d3a45da9d90ee7d50e5184e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51e1565c5d3a45da9d90ee7d50e5184e, State: Initialized, Role: FOLLOWER
I20260620 18:39:19.822382 6183 consensus_queue.cc:260] T c99c11c996b140d3aaa884c41fabe2e3 P 51e1565c5d3a45da9d90ee7d50e5184e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51e1565c5d3a45da9d90ee7d50e5184e" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35831 } } peers { permanent_uuid: "129d88bd84a241a68291203014b805d0" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 42741 } } peers { permanent_uuid: "c995f8a8741a4540b650832e93ff4d36" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 36829 } }
I20260620 18:39:19.827284 6184 raft_consensus.cc:348] T c99c11c996b140d3aaa884c41fabe2e3 P 129d88bd84a241a68291203014b805d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51e1565c5d3a45da9d90ee7d50e5184e" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35831 } } peers { permanent_uuid: "129d88bd84a241a68291203014b805d0" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 42741 } } peers { permanent_uuid: "c995f8a8741a4540b650832e93ff4d36" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 36829 } }
I20260620 18:39:19.827286 6185 raft_consensus.cc:348] T c99c11c996b140d3aaa884c41fabe2e3 P c995f8a8741a4540b650832e93ff4d36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51e1565c5d3a45da9d90ee7d50e5184e" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35831 } } peers { permanent_uuid: "129d88bd84a241a68291203014b805d0" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 42741 } } peers { permanent_uuid: "c995f8a8741a4540b650832e93ff4d36" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 36829 } }
I20260620 18:39:19.827745 6185 raft_consensus.cc:374] T c99c11c996b140d3aaa884c41fabe2e3 P c995f8a8741a4540b650832e93ff4d36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:39:19.827749 6184 raft_consensus.cc:374] T c99c11c996b140d3aaa884c41fabe2e3 P 129d88bd84a241a68291203014b805d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:39:19.827849 6184 raft_consensus.cc:729] T c99c11c996b140d3aaa884c41fabe2e3 P 129d88bd84a241a68291203014b805d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 129d88bd84a241a68291203014b805d0, State: Initialized, Role: FOLLOWER
I20260620 18:39:19.827929 6185 raft_consensus.cc:729] T c99c11c996b140d3aaa884c41fabe2e3 P c995f8a8741a4540b650832e93ff4d36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c995f8a8741a4540b650832e93ff4d36, State: Initialized, Role: FOLLOWER
I20260620 18:39:19.828529 6165 heartbeater.cc:503] Master 127.22.37.62:39147 was elected leader, sending a full tablet report...
I20260620 18:39:19.828665 6184 consensus_queue.cc:260] T c99c11c996b140d3aaa884c41fabe2e3 P 129d88bd84a241a68291203014b805d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51e1565c5d3a45da9d90ee7d50e5184e" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35831 } } peers { permanent_uuid: "129d88bd84a241a68291203014b805d0" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 42741 } } peers { permanent_uuid: "c995f8a8741a4540b650832e93ff4d36" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 36829 } }
I20260620 18:39:19.829231 6183 ts_tablet_manager.cc:1428] T c99c11c996b140d3aaa884c41fabe2e3 P 51e1565c5d3a45da9d90ee7d50e5184e: Time spent starting tablet: real 0.016s user 0.012s sys 0.005s
I20260620 18:39:19.829056 6185 consensus_queue.cc:260] T c99c11c996b140d3aaa884c41fabe2e3 P c995f8a8741a4540b650832e93ff4d36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51e1565c5d3a45da9d90ee7d50e5184e" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35831 } } peers { permanent_uuid: "129d88bd84a241a68291203014b805d0" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 42741 } } peers { permanent_uuid: "c995f8a8741a4540b650832e93ff4d36" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 36829 } }
I20260620 18:39:19.830319 6032 heartbeater.cc:503] Master 127.22.37.62:39147 was elected leader, sending a full tablet report...
I20260620 18:39:19.831029 6185 ts_tablet_manager.cc:1428] T c99c11c996b140d3aaa884c41fabe2e3 P c995f8a8741a4540b650832e93ff4d36: Time spent starting tablet: real 0.015s user 0.014s sys 0.001s
I20260620 18:39:19.832654 6184 ts_tablet_manager.cc:1428] T c99c11c996b140d3aaa884c41fabe2e3 P 129d88bd84a241a68291203014b805d0: Time spent starting tablet: real 0.016s user 0.011s sys 0.005s
W20260620 18:39:19.911458 5899 tablet.cc:2369] T c99c11c996b140d3aaa884c41fabe2e3 P 129d88bd84a241a68291203014b805d0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:39:19.921967 6033 tablet.cc:2369] T c99c11c996b140d3aaa884c41fabe2e3 P c995f8a8741a4540b650832e93ff4d36: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:39:19.936638 6166 tablet.cc:2369] T c99c11c996b140d3aaa884c41fabe2e3 P 51e1565c5d3a45da9d90ee7d50e5184e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:39:20.064481 6191 raft_consensus.cc:482] T c99c11c996b140d3aaa884c41fabe2e3 P c995f8a8741a4540b650832e93ff4d36 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:39:20.064881 6191 raft_consensus.cc:504] T c99c11c996b140d3aaa884c41fabe2e3 P c995f8a8741a4540b650832e93ff4d36 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51e1565c5d3a45da9d90ee7d50e5184e" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35831 } } peers { permanent_uuid: "129d88bd84a241a68291203014b805d0" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 42741 } } peers { permanent_uuid: "c995f8a8741a4540b650832e93ff4d36" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 36829 } }
I20260620 18:39:20.066792 6191 leader_election.cc:288] T c99c11c996b140d3aaa884c41fabe2e3 P c995f8a8741a4540b650832e93ff4d36 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51e1565c5d3a45da9d90ee7d50e5184e (127.22.37.3:35831), 129d88bd84a241a68291203014b805d0 (127.22.37.1:42741)
I20260620 18:39:20.073101 6179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:20.067427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.2:56679 (local address 127.22.37.3:35831)
0620 18:39:20.067637 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:20.067644 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:20.067871 (+ 227us) server_negotiation.cc:407] Connection header received
0620 18:39:20.068152 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:20.068158 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:20.068227 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:20.068321 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:39:20.068861 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:20.069633 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:20.071401 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:20.072133 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:20.072278 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:20.072403 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:20.072695 (+ 292us) server_negotiation.cc:299] Negotiation successful
0620 18:39:20.072825 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 18:39:20.073261 6192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:20.066979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.3:35831 (local address 127.22.37.2:56679)
0620 18:39:20.067764 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:20.067780 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:20.068009 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:20.068340 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:20.068345 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:20.068370 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:39:20.068722 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:20.068732 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:20.069777 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:20.069785 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:20.071260 (+ 1475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:20.071271 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:20.072337 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:20.072343 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:20.072524 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:20.072550 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 18:39:20.072668 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 18:39:20.072799 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"mutex_wait_us":26,"thread_start_us":471,"threads_started":1}
I20260620 18:39:20.073292 6181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:20.067204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.2:41727 (local address 127.22.37.1:42741)
0620 18:39:20.067650 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:20.067659 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:20.068217 (+ 558us) server_negotiation.cc:407] Connection header received
0620 18:39:20.068443 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:20.068448 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:20.068531 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:20.068629 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:39:20.069193 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:20.069945 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:20.071582 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:20.072332 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:20.072498 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:20.072603 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:20.072905 (+ 302us) server_negotiation.cc:299] Negotiation successful
0620 18:39:20.073025 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289}
I20260620 18:39:20.073441 6193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:20.067138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.1:42741 (local address 127.22.37.2:41727)
0620 18:39:20.068114 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:20.068127 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:20.068269 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:20.068643 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:20.068651 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:20.068686 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:39:20.069032 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:20.069040 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:20.070079 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:20.070084 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:20.071440 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:20.071450 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:20.072565 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:20.072570 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:20.072740 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:20.072762 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 18:39:20.072872 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 18:39:20.072990 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":875,"mutex_wait_us":27,"thread_start_us":613,"threads_started":1}
I20260620 18:39:20.074458 6120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c99c11c996b140d3aaa884c41fabe2e3" candidate_uuid: "c995f8a8741a4540b650832e93ff4d36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51e1565c5d3a45da9d90ee7d50e5184e" is_pre_election: true
I20260620 18:39:20.074452 5853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c99c11c996b140d3aaa884c41fabe2e3" candidate_uuid: "c995f8a8741a4540b650832e93ff4d36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "129d88bd84a241a68291203014b805d0" is_pre_election: true
I20260620 18:39:20.074966 6120 raft_consensus.cc:2454] T c99c11c996b140d3aaa884c41fabe2e3 P 51e1565c5d3a45da9d90ee7d50e5184e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c995f8a8741a4540b650832e93ff4d36 in term 0.
I20260620 18:39:20.074975 5853 raft_consensus.cc:2454] T c99c11c996b140d3aaa884c41fabe2e3 P 129d88bd84a241a68291203014b805d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c995f8a8741a4540b650832e93ff4d36 in term 0.
I20260620 18:39:20.075620 5922 leader_election.cc:302] T c99c11c996b140d3aaa884c41fabe2e3 P c995f8a8741a4540b650832e93ff4d36 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 129d88bd84a241a68291203014b805d0, c995f8a8741a4540b650832e93ff4d36; no voters:
I20260620 18:39:20.076027 6191 raft_consensus.cc:2781] T c99c11c996b140d3aaa884c41fabe2e3 P c995f8a8741a4540b650832e93ff4d36 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:39:20.076185 6191 raft_consensus.cc:482] T c99c11c996b140d3aaa884c41fabe2e3 P c995f8a8741a4540b650832e93ff4d36 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:39:20.076299 6191 raft_consensus.cc:3037] T c99c11c996b140d3aaa884c41fabe2e3 P c995f8a8741a4540b650832e93ff4d36 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:39:20.078483 6191 raft_consensus.cc:504] T c99c11c996b140d3aaa884c41fabe2e3 P c995f8a8741a4540b650832e93ff4d36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51e1565c5d3a45da9d90ee7d50e5184e" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35831 } } peers { permanent_uuid: "129d88bd84a241a68291203014b805d0" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 42741 } } peers { permanent_uuid: "c995f8a8741a4540b650832e93ff4d36" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 36829 } }
I20260620 18:39:20.079056 6191 leader_election.cc:288] T c99c11c996b140d3aaa884c41fabe2e3 P c995f8a8741a4540b650832e93ff4d36 [CANDIDATE]: Term 1 election: Requested vote from peers 51e1565c5d3a45da9d90ee7d50e5184e (127.22.37.3:35831), 129d88bd84a241a68291203014b805d0 (127.22.37.1:42741)
I20260620 18:39:20.079588 5853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c99c11c996b140d3aaa884c41fabe2e3" candidate_uuid: "c995f8a8741a4540b650832e93ff4d36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "129d88bd84a241a68291203014b805d0"
I20260620 18:39:20.079809 5853 raft_consensus.cc:3037] T c99c11c996b140d3aaa884c41fabe2e3 P 129d88bd84a241a68291203014b805d0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:39:20.079908 6120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c99c11c996b140d3aaa884c41fabe2e3" candidate_uuid: "c995f8a8741a4540b650832e93ff4d36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51e1565c5d3a45da9d90ee7d50e5184e"
I20260620 18:39:20.080127 6120 raft_consensus.cc:3037] T c99c11c996b140d3aaa884c41fabe2e3 P 51e1565c5d3a45da9d90ee7d50e5184e [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:39:20.082302 6120 raft_consensus.cc:2454] T c99c11c996b140d3aaa884c41fabe2e3 P 51e1565c5d3a45da9d90ee7d50e5184e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c995f8a8741a4540b650832e93ff4d36 in term 1.
I20260620 18:39:20.082326 5853 raft_consensus.cc:2454] T c99c11c996b140d3aaa884c41fabe2e3 P 129d88bd84a241a68291203014b805d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c995f8a8741a4540b650832e93ff4d36 in term 1.
I20260620 18:39:20.082769 5920 leader_election.cc:302] T c99c11c996b140d3aaa884c41fabe2e3 P c995f8a8741a4540b650832e93ff4d36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51e1565c5d3a45da9d90ee7d50e5184e, c995f8a8741a4540b650832e93ff4d36; no voters:
I20260620 18:39:20.083112 6191 raft_consensus.cc:2781] T c99c11c996b140d3aaa884c41fabe2e3 P c995f8a8741a4540b650832e93ff4d36 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:39:20.083545 6191 raft_consensus.cc:686] T c99c11c996b140d3aaa884c41fabe2e3 P c995f8a8741a4540b650832e93ff4d36 [term 1 LEADER]: Becoming Leader. State: Replica: c995f8a8741a4540b650832e93ff4d36, State: Running, Role: LEADER
I20260620 18:39:20.083998 6191 consensus_queue.cc:237] T c99c11c996b140d3aaa884c41fabe2e3 P c995f8a8741a4540b650832e93ff4d36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51e1565c5d3a45da9d90ee7d50e5184e" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35831 } } peers { permanent_uuid: "129d88bd84a241a68291203014b805d0" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 42741 } } peers { permanent_uuid: "c995f8a8741a4540b650832e93ff4d36" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 36829 } }
I20260620 18:39:20.088696 5709 catalog_manager.cc:5495] T c99c11c996b140d3aaa884c41fabe2e3 P c995f8a8741a4540b650832e93ff4d36 reported cstate change: term changed from 0 to 1, leader changed from <none> to c995f8a8741a4540b650832e93ff4d36 (127.22.37.2). New cstate: current_term: 1 leader_uuid: "c995f8a8741a4540b650832e93ff4d36" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51e1565c5d3a45da9d90ee7d50e5184e" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "129d88bd84a241a68291203014b805d0" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 42741 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c995f8a8741a4540b650832e93ff4d36" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 36829 } health_report { overall_health: HEALTHY } } }
I20260620 18:39:20.150283 6182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:20.144461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41504 (local address 127.22.37.2:36829)
0620 18:39:20.144675 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:20.144679 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:20.144697 (+ 18us) server_negotiation.cc:407] Connection header received
0620 18:39:20.144878 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:20.144882 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:20.144960 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:20.145086 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:39:20.145514 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:20.146201 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:20.148089 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:20.148515 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:20.148609 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:20.149061 (+ 452us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:39:20.149643 (+ 582us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:39:20.149753 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:20.149893 (+ 140us) server_negotiation.cc:299] Negotiation successful
0620 18:39:20.149960 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 18:39:20.158560 6120 raft_consensus.cc:1261] T c99c11c996b140d3aaa884c41fabe2e3 P 51e1565c5d3a45da9d90ee7d50e5184e [term 1 FOLLOWER]: Refusing update from remote peer c995f8a8741a4540b650832e93ff4d36: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:39:20.158560 5853 raft_consensus.cc:1261] T c99c11c996b140d3aaa884c41fabe2e3 P 129d88bd84a241a68291203014b805d0 [term 1 FOLLOWER]: Refusing update from remote peer c995f8a8741a4540b650832e93ff4d36: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:39:20.159547 6194 consensus_queue.cc:1035] T c99c11c996b140d3aaa884c41fabe2e3 P c995f8a8741a4540b650832e93ff4d36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "129d88bd84a241a68291203014b805d0" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 42741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:39:20.159914 6191 consensus_queue.cc:1035] T c99c11c996b140d3aaa884c41fabe2e3 P c995f8a8741a4540b650832e93ff4d36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51e1565c5d3a45da9d90ee7d50e5184e" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:39:20.193439 5708 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57896:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:39:20.193746 5708 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57896:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:39:20.198143 5708 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a593d764462a463d8334200095758268: Sending DeleteTablet for 3 replicas of tablet c99c11c996b140d3aaa884c41fabe2e3
I20260620 18:39:20.199415 5967 tablet_service.cc:1514] Processing DeleteTablet for tablet c99c11c996b140d3aaa884c41fabe2e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:39:20 UTC) from {username='kudu', principal='kudu/127.22.37.62@KRBTEST.COM'} at 127.0.0.1:41496
I20260620 18:39:20.200150 6100 tablet_service.cc:1514] Processing DeleteTablet for tablet c99c11c996b140d3aaa884c41fabe2e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:39:20 UTC) from {username='kudu', principal='kudu/127.22.37.62@KRBTEST.COM'} at 127.0.0.1:36178
I20260620 18:39:20.200527 6207 tablet_replica.cc:330] T c99c11c996b140d3aaa884c41fabe2e3 P c995f8a8741a4540b650832e93ff4d36: stopping tablet replica
I20260620 18:39:20.201103 6207 raft_consensus.cc:2229] T c99c11c996b140d3aaa884c41fabe2e3 P c995f8a8741a4540b650832e93ff4d36 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:39:20.201449 5833 tablet_service.cc:1514] Processing DeleteTablet for tablet c99c11c996b140d3aaa884c41fabe2e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:39:20 UTC) from {username='kudu', principal='kudu/127.22.37.62@KRBTEST.COM'} at 127.0.0.1:52132
I20260620 18:39:20.201918 6207 raft_consensus.cc:2258] T c99c11c996b140d3aaa884c41fabe2e3 P c995f8a8741a4540b650832e93ff4d36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:39:20.202132 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 5770
I20260620 18:39:20.204329 6207 ts_tablet_manager.cc:1905] T c99c11c996b140d3aaa884c41fabe2e3 P c995f8a8741a4540b650832e93ff4d36: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:39:20.210323 6208 tablet_replica.cc:330] T c99c11c996b140d3aaa884c41fabe2e3 P 51e1565c5d3a45da9d90ee7d50e5184e: stopping tablet replica
I20260620 18:39:20.210992 6208 raft_consensus.cc:2229] T c99c11c996b140d3aaa884c41fabe2e3 P 51e1565c5d3a45da9d90ee7d50e5184e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:39:20.211688 6208 raft_consensus.cc:2258] T c99c11c996b140d3aaa884c41fabe2e3 P 51e1565c5d3a45da9d90ee7d50e5184e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:39:20.212371 6209 tablet_replica.cc:330] T c99c11c996b140d3aaa884c41fabe2e3 P 129d88bd84a241a68291203014b805d0: stopping tablet replica
I20260620 18:39:20.212909 6207 ts_tablet_manager.cc:1918] T c99c11c996b140d3aaa884c41fabe2e3 P c995f8a8741a4540b650832e93ff4d36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:39:20.213160 6207 log.cc:1196] T c99c11c996b140d3aaa884c41fabe2e3 P c995f8a8741a4540b650832e93ff4d36: Deleting WAL directory at /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-1/wal/wals/c99c11c996b140d3aaa884c41fabe2e3
I20260620 18:39:20.213290 6209 raft_consensus.cc:2229] T c99c11c996b140d3aaa884c41fabe2e3 P 129d88bd84a241a68291203014b805d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:39:20.213762 6207 ts_tablet_manager.cc:1939] T c99c11c996b140d3aaa884c41fabe2e3 P c995f8a8741a4540b650832e93ff4d36: Deleting consensus metadata
I20260620 18:39:20.214079 6209 raft_consensus.cc:2258] T c99c11c996b140d3aaa884c41fabe2e3 P 129d88bd84a241a68291203014b805d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:39:20.214951 6208 ts_tablet_manager.cc:1905] T c99c11c996b140d3aaa884c41fabe2e3 P 51e1565c5d3a45da9d90ee7d50e5184e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:39:20.215736 5694 catalog_manager.cc:4841] TS c995f8a8741a4540b650832e93ff4d36 (127.22.37.2:36829): tablet c99c11c996b140d3aaa884c41fabe2e3 (table test-table [id=7b23d07f173346e6811c590def77d78c]) successfully deleted
I20260620 18:39:20.216732 6209 ts_tablet_manager.cc:1905] T c99c11c996b140d3aaa884c41fabe2e3 P 129d88bd84a241a68291203014b805d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:39:20.220465 6208 ts_tablet_manager.cc:1918] T c99c11c996b140d3aaa884c41fabe2e3 P 51e1565c5d3a45da9d90ee7d50e5184e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:39:20.220671 6208 log.cc:1196] T c99c11c996b140d3aaa884c41fabe2e3 P 51e1565c5d3a45da9d90ee7d50e5184e: Deleting WAL directory at /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-2/wal/wals/c99c11c996b140d3aaa884c41fabe2e3
I20260620 18:39:20.221140 6208 ts_tablet_manager.cc:1939] T c99c11c996b140d3aaa884c41fabe2e3 P 51e1565c5d3a45da9d90ee7d50e5184e: Deleting consensus metadata
I20260620 18:39:20.224085 5693 catalog_manager.cc:4841] TS 51e1565c5d3a45da9d90ee7d50e5184e (127.22.37.3:35831): tablet c99c11c996b140d3aaa884c41fabe2e3 (table test-table [id=7b23d07f173346e6811c590def77d78c]) successfully deleted
I20260620 18:39:20.225148 6209 ts_tablet_manager.cc:1918] T c99c11c996b140d3aaa884c41fabe2e3 P 129d88bd84a241a68291203014b805d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:39:20.225353 6209 log.cc:1196] T c99c11c996b140d3aaa884c41fabe2e3 P 129d88bd84a241a68291203014b805d0: Deleting WAL directory at /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-0/wal/wals/c99c11c996b140d3aaa884c41fabe2e3
I20260620 18:39:20.225816 6209 ts_tablet_manager.cc:1939] T c99c11c996b140d3aaa884c41fabe2e3 P 129d88bd84a241a68291203014b805d0: Deleting consensus metadata
I20260620 18:39:20.227660 5695 catalog_manager.cc:4841] TS 129d88bd84a241a68291203014b805d0 (127.22.37.1:42741): tablet c99c11c996b140d3aaa884c41fabe2e3 (table test-table [id=7b23d07f173346e6811c590def77d78c]) successfully deleted
Jun 20 18:39:20 dist-test-slave-5lgp krb5kdc[5664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980757, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.37.1@KRBTEST.COM
I20260620 18:39:20.240298 6181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:20.203258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52148 (local address 127.22.37.1:42741)
0620 18:39:20.203432 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:20.203438 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:20.222072 (+ 18634us) server_negotiation.cc:407] Connection header received
0620 18:39:20.222287 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:20.222300 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:20.222395 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:20.222512 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:20.224743 (+ 2231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:20.225991 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:20.227998 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:20.228701 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:20.228851 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:20.233237 (+ 4386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:20.233254 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:20.233275 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:20.233322 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:20.237182 (+ 3860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:20.237987 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:20.237998 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:20.238002 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:20.238184 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:20.238559 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:20.238566 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:20.238570 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:20.239072 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:20.239389 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:20.239616 (+ 227us) server_negotiation.cc:299] Negotiation successful
0620 18:39:20.239860 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 18:39:20.241020 5893 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:39:20.331650 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 5770
I20260620 18:39:20.349138 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 5904
Jun 20 18:39:20 dist-test-slave-5lgp krb5kdc[5664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980757, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.37.2@KRBTEST.COM
I20260620 18:39:20.367267 6182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:20.350102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41514 (local address 127.22.37.2:36829)
0620 18:39:20.350277 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:20.350281 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:20.350299 (+ 18us) server_negotiation.cc:407] Connection header received
0620 18:39:20.350444 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:20.350447 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:20.350512 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:20.350593 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:20.352382 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:20.353200 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:20.355274 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:20.355680 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:20.355769 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:20.361201 (+ 5432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:20.361215 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:20.361230 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:20.361270 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:20.364396 (+ 3126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:20.365096 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:20.365104 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:20.365106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:20.365236 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:20.365625 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:20.365631 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:20.365634 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:20.366013 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:20.366559 (+ 546us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:20.366717 (+ 158us) server_negotiation.cc:299] Negotiation successful
0620 18:39:20.366945 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 18:39:20.367966 6027 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:39:20.481130 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 5904
I20260620 18:39:20.499815 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 6037
Jun 20 18:39:20 dist-test-slave-5lgp krb5kdc[5664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980757, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.37.3@KRBTEST.COM
I20260620 18:39:20.516311 6179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:20.500628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36194 (local address 127.22.37.3:35831)
0620 18:39:20.500832 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:20.500841 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:20.500953 (+ 112us) server_negotiation.cc:407] Connection header received
0620 18:39:20.501206 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:20.501212 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:20.501285 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:20.501422 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:20.502952 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:20.503746 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:20.505219 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:20.505715 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:20.505871 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:20.510716 (+ 4845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:20.510731 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:20.510749 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:20.510792 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:20.513640 (+ 2848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:20.514341 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:20.514349 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:20.514352 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:20.514515 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:20.514865 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:20.514868 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:20.514870 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:20.515304 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:20.515518 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:20.515831 (+ 313us) server_negotiation.cc:299] Negotiation successful
0620 18:39:20.516000 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 18:39:20.516978 6160 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:39:20.607748 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 6037
I20260620 18:39:20.625150 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 5677
I20260620 18:39:20.626292 5739 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:39:20.736691 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 5677
I20260620 18:39:40.758836 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.62:39147
--webserver_interface=127.22.37.62
--webserver_port=34685
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab
--principal=kudu/127.22.37.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:33907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.37.62:39147
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:39:41.097045 6215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:41.097424 6215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:41.097560 6215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:41.105046 6215 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 18:39:41.105173 6215 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 18:39:41.105219 6215 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 18:39:41.105271 6215 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:39:41.105333 6215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:41.105499 6215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:41.105561 6215 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:39:41.105633 6215 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:39:41.115901 6215 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:33907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.22.37.62:39147
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab
--principal=kudu/127.22.37.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.62:39147
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.37.62
--webserver_port=34685
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:39:41.117230 6215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:41.118836 6215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:39:41.143914 6221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:41.144052 6222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:41.144047 6224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:41.144992 6215 server_base.cc:1034] running on GCE node
I20260620 18:39:41.145942 6215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:41.147559 6215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:41.148854 6215 hybrid_clock.cc:648] HybridClock initialized: now 1781980781148759 us; error 55 us; skew 500 ppm
I20260620 18:39:41.149294 6215 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:39:41 dist-test-slave-5lgp krb5kdc[5664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980781, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:39:41.152892 6215 init.cc:377] Logged in from keytab as kudu/127.22.37.62@KRBTEST.COM (short username kudu)
I20260620 18:39:41.154525 6215 webserver.cc:460] Webserver started at http://127.22.37.62:34685/ using document root <none> and password file <none>
I20260620 18:39:41.155328 6215 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:41.155479 6215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:41.161427 6215 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260620 18:39:41.164346 6231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:41.165748 6215 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:39:41.165964 6215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/master-0/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/master-0/wal
uuid: "a593d764462a463d8334200095758268"
format_stamp: "Formatted at 2026-06-20 18:39:17 on dist-test-slave-5lgp"
server_key: "9262f4c61f5b39011a2d4c9108445344"
server_key_iv: "f62851751da673374e04ecb83bbba083"
server_key_version: "encryptionkey@0"
I20260620 18:39:41.167083 6215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:41.205022 6215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:41.210121 6215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:41.210457 6215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:41.230352 6215 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.62:39147
I20260620 18:39:41.230389 6282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.62:39147 every 8 connection(s)
I20260620 18:39:41.231951 6215 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
I20260620 18:39:41.233058 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 6215
I20260620 18:39:41.234127 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.37.1:42741
--local_ip_for_outbound_sockets=127.22.37.1
--tserver_master_addrs=127.22.37.62:39147
--webserver_port=46361
--webserver_interface=127.22.37.1
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.1.keytab
--principal=kudu/127.22.37.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:33907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:39:41.239989 6283 sys_catalog.cc:263] Verifying existing consensus state
I20260620 18:39:41.244742 6283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a593d764462a463d8334200095758268: Bootstrap starting.
I20260620 18:39:41.274837 6283 log.cc:824] T 00000000000000000000000000000000 P a593d764462a463d8334200095758268: Log is configured to *not* fsync() on all Append() calls
I20260620 18:39:41.290467 6283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a593d764462a463d8334200095758268: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 18:39:41.291167 6283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a593d764462a463d8334200095758268: Bootstrap complete.
I20260620 18:39:41.306671 6283 raft_consensus.cc:348] T 00000000000000000000000000000000 P a593d764462a463d8334200095758268 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a593d764462a463d8334200095758268" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 39147 } }
I20260620 18:39:41.308013 6283 raft_consensus.cc:729] T 00000000000000000000000000000000 P a593d764462a463d8334200095758268 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a593d764462a463d8334200095758268, State: Initialized, Role: FOLLOWER
I20260620 18:39:41.308779 6283 consensus_queue.cc:260] T 00000000000000000000000000000000 P a593d764462a463d8334200095758268 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a593d764462a463d8334200095758268" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 39147 } }
I20260620 18:39:41.309064 6283 raft_consensus.cc:388] T 00000000000000000000000000000000 P a593d764462a463d8334200095758268 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:39:41.309187 6283 raft_consensus.cc:482] T 00000000000000000000000000000000 P a593d764462a463d8334200095758268 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:39:41.309399 6283 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a593d764462a463d8334200095758268 [term 1 FOLLOWER]: Advancing to term 2
I20260620 18:39:41.315088 6283 raft_consensus.cc:504] T 00000000000000000000000000000000 P a593d764462a463d8334200095758268 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a593d764462a463d8334200095758268" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 39147 } }
I20260620 18:39:41.315744 6283 leader_election.cc:302] T 00000000000000000000000000000000 P a593d764462a463d8334200095758268 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a593d764462a463d8334200095758268; no voters:
I20260620 18:39:41.316432 6283 leader_election.cc:288] T 00000000000000000000000000000000 P a593d764462a463d8334200095758268 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 18:39:41.316606 6287 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a593d764462a463d8334200095758268 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 18:39:41.318221 6287 raft_consensus.cc:686] T 00000000000000000000000000000000 P a593d764462a463d8334200095758268 [term 2 LEADER]: Becoming Leader. State: Replica: a593d764462a463d8334200095758268, State: Running, Role: LEADER
I20260620 18:39:41.318831 6283 sys_catalog.cc:564] T 00000000000000000000000000000000 P a593d764462a463d8334200095758268 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:39:41.318914 6287 consensus_queue.cc:237] T 00000000000000000000000000000000 P a593d764462a463d8334200095758268 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a593d764462a463d8334200095758268" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 39147 } }
I20260620 18:39:41.321590 6289 sys_catalog.cc:455] T 00000000000000000000000000000000 P a593d764462a463d8334200095758268 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a593d764462a463d8334200095758268. Latest consensus state: current_term: 2 leader_uuid: "a593d764462a463d8334200095758268" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a593d764462a463d8334200095758268" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 39147 } } }
I20260620 18:39:41.321973 6289 sys_catalog.cc:458] T 00000000000000000000000000000000 P a593d764462a463d8334200095758268 [sys.catalog]: This master's current role is: LEADER
I20260620 18:39:41.322479 6288 sys_catalog.cc:455] T 00000000000000000000000000000000 P a593d764462a463d8334200095758268 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a593d764462a463d8334200095758268" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a593d764462a463d8334200095758268" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 39147 } } }
I20260620 18:39:41.322788 6288 sys_catalog.cc:458] T 00000000000000000000000000000000 P a593d764462a463d8334200095758268 [sys.catalog]: This master's current role is: LEADER
I20260620 18:39:41.334311 6294 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:39:41.343458 6294 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:39:41.344982 6294 catalog_manager.cc:1259] Loaded cluster ID: a5cfbde2c9ad4e62b179da5e8af50d12
I20260620 18:39:41.345144 6294 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:39:41.349399 6294 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:39:41.350246 6304 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 18:39:41.351261 6294 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a593d764462a463d8334200095758268: Loaded TSK: 0
I20260620 18:39:41.386252 6294 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a593d764462a463d8334200095758268: Generated new TSK 1
I20260620 18:39:41.386642 6294 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 18:39:41.604147 6285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:41.604502 6285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:41.604668 6285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:41.612059 6285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:41.612215 6285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:41.612396 6285 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.37.1
I20260620 18:39:41.622509 6285 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:33907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.1.keytab
--principal=kudu/127.22.37.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.37.1:42741
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.37.1
--webserver_port=46361
--tserver_master_addrs=127.22.37.62:39147
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.37.1
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:39:41.623761 6285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:41.625070 6285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:39:41.640690 6311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:41.641230 6313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:41.641775 6310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:41.641737 6285 server_base.cc:1034] running on GCE node
I20260620 18:39:41.642650 6285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:41.643954 6285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:41.645231 6285 hybrid_clock.cc:648] HybridClock initialized: now 1781980781645139 us; error 93 us; skew 500 ppm
I20260620 18:39:41.645696 6285 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:39:41 dist-test-slave-5lgp krb5kdc[5664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980781, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:39:41.649976 6285 init.cc:377] Logged in from keytab as kudu/127.22.37.1@KRBTEST.COM (short username kudu)
I20260620 18:39:41.651911 6285 webserver.cc:460] Webserver started at http://127.22.37.1:46361/ using document root <none> and password file <none>
I20260620 18:39:41.652832 6285 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:41.652993 6285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:41.660607 6285 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 18:39:41.664281 6320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:41.666038 6285 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:39:41.666236 6285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-0/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-0/wal
uuid: "129d88bd84a241a68291203014b805d0"
format_stamp: "Formatted at 2026-06-20 18:39:18 on dist-test-slave-5lgp"
server_key: "5fa8a0c42fb597c961965ef437c4e92a"
server_key_iv: "a199ce8a6a32e88fd610debef841e3df"
server_key_version: "encryptionkey@0"
I20260620 18:39:41.667743 6285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:41.700083 6285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:41.705655 6285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:41.705965 6285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:41.707122 6285 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:39:41.708647 6285 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:39:41.708735 6285 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:41.708897 6285 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:39:41.708961 6285 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:41.751351 6285 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.1:42741
I20260620 18:39:41.751444 6432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.1:42741 every 8 connection(s)
I20260620 18:39:41.752789 6285 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-0/data/info.pb
I20260620 18:39:41.762876 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 6285
I20260620 18:39:41.763496 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.37.2:36829
--local_ip_for_outbound_sockets=127.22.37.2
--tserver_master_addrs=127.22.37.62:39147
--webserver_port=44147
--webserver_interface=127.22.37.2
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.2.keytab
--principal=kudu/127.22.37.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:33907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 18:39:41 dist-test-slave-5lgp krb5kdc[5664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980781, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.1@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM
I20260620 18:39:41.776646 6435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:41.755663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.1:60235 (local address 127.22.37.62:39147)
0620 18:39:41.756313 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:41.756324 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:41.756816 (+ 492us) server_negotiation.cc:407] Connection header received
0620 18:39:41.758965 (+ 2149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:41.758978 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:41.759288 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:41.759569 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:41.760954 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:41.762241 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:41.763842 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:41.764996 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:41.765186 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:41.768538 (+ 3352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:41.768546 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:41.768560 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:41.768601 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:41.772046 (+ 3445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:41.772866 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:41.772878 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:41.772889 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:41.773051 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:41.773623 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:41.773629 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:41.773632 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:41.774237 (+ 605us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:41.774386 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:41.775291 (+ 905us) server_negotiation.cc:299] Negotiation successful
0620 18:39:41.775613 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":304,"threads_started":1}
I20260620 18:39:41.776997 6436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:41.755849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:39147 (local address 127.22.37.1:60235)
0620 18:39:41.756645 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:41.756697 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:41.758489 (+ 1792us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:41.759716 (+ 1227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:41.759723 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:41.760210 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:41.760743 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:41.760761 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:41.762389 (+ 1628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:41.762392 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:41.763682 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:41.763697 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:41.765208 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:41.765213 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:41.765453 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:41.766147 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:41.766172 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:41.768270 (+ 2098us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:39:41.772210 (+ 3940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:41.772217 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:41.772228 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:41.772653 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:41.773165 (+ 512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:41.773173 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:41.773179 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:41.773456 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:41.774375 (+ 919us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:39:41.774386 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:39:41.774812 (+ 426us) client_negotiation.cc:769] Sending connection context
0620 18:39:41.775098 (+ 286us) client_negotiation.cc:240] Negotiation successful
0620 18:39:41.775687 (+ 589us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":332,"threads_started":1}
I20260620 18:39:41.779946 6433 heartbeater.cc:347] Connected to a master server at 127.22.37.62:39147
I20260620 18:39:41.780282 6433 heartbeater.cc:464] Registering TS with master...
I20260620 18:39:41.781096 6433 heartbeater.cc:511] Master 127.22.37.62:39147 requested a full tablet report, sending...
I20260620 18:39:41.783457 6248 ts_manager.cc:194] Registered new tserver with Master: 129d88bd84a241a68291203014b805d0 (127.22.37.1:42741)
I20260620 18:39:41.784884 6248 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.37.1@KRBTEST.COM'} at 127.22.37.1:60235
W20260620 18:39:42.096699 6437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:42.097016 6437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:42.097167 6437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:42.104615 6437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:42.104741 6437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:42.104981 6437 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.37.2
I20260620 18:39:42.114889 6437 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:33907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.2.keytab
--principal=kudu/127.22.37.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.37.2:36829
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.37.2
--webserver_port=44147
--tserver_master_addrs=127.22.37.62:39147
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.37.2
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:39:42.116218 6437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:42.117607 6437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:39:42.130749 6442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:42.130805 6443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:42.130884 6445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:42.131963 6437 server_base.cc:1034] running on GCE node
I20260620 18:39:42.132507 6437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:42.133440 6437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:42.134639 6437 hybrid_clock.cc:648] HybridClock initialized: now 1781980782134574 us; error 57 us; skew 500 ppm
I20260620 18:39:42.134927 6437 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:39:42 dist-test-slave-5lgp krb5kdc[5664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980782, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:39:42.138137 6437 init.cc:377] Logged in from keytab as kudu/127.22.37.2@KRBTEST.COM (short username kudu)
I20260620 18:39:42.139622 6437 webserver.cc:460] Webserver started at http://127.22.37.2:44147/ using document root <none> and password file <none>
I20260620 18:39:42.140334 6437 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:42.140499 6437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:42.145829 6437 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 18:39:42.148414 6452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:42.149842 6437 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:39:42.149966 6437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-1/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-1/wal
uuid: "c995f8a8741a4540b650832e93ff4d36"
format_stamp: "Formatted at 2026-06-20 18:39:19 on dist-test-slave-5lgp"
server_key: "1125f2023af9f5c615e75957e6bcb2aa"
server_key_iv: "e593df8bfc77c25c10e5285125c17478"
server_key_version: "encryptionkey@0"
I20260620 18:39:42.150907 6437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:42.165227 6437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:42.168905 6437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:42.169112 6437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:42.170028 6437 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:39:42.171300 6437 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:39:42.171382 6437 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:42.171484 6437 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:39:42.171540 6437 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:42.203334 6437 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.2:36829
I20260620 18:39:42.203392 6564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.2:36829 every 8 connection(s)
I20260620 18:39:42.204843 6437 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-1/data/info.pb
I20260620 18:39:42.212108 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 6437
I20260620 18:39:42.212853 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.37.3:35831
--local_ip_for_outbound_sockets=127.22.37.3
--tserver_master_addrs=127.22.37.62:39147
--webserver_port=36447
--webserver_interface=127.22.37.3
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.3.keytab
--principal=kudu/127.22.37.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:33907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 18:39:42 dist-test-slave-5lgp krb5kdc[5664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980782, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.2@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM
I20260620 18:39:42.225972 6435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:42.207404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.2:43903 (local address 127.22.37.62:39147)
0620 18:39:42.207630 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:42.207635 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:42.208752 (+ 1117us) server_negotiation.cc:407] Connection header received
0620 18:39:42.210572 (+ 1820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:42.210577 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:42.210642 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:42.210741 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:42.212158 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:42.213226 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:42.215084 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:42.215818 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:42.215959 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:42.219740 (+ 3781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:42.219750 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:42.219758 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:42.219815 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:42.222687 (+ 2872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:42.223600 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:42.223604 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:42.223607 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:42.223732 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:42.224213 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:42.224217 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:42.224219 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:42.224475 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:42.224581 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:42.225549 (+ 968us) server_negotiation.cc:299] Negotiation successful
0620 18:39:42.225724 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260620 18:39:42.227533 6567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:42.207858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:39147 (local address 127.22.37.2:43903)
0620 18:39:42.208591 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:42.208630 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:42.210332 (+ 1702us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:42.210944 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:42.210951 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:42.211448 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:42.211989 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:42.212005 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:42.213477 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:42.213482 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:42.214931 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:42.214944 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:42.216020 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:42.216024 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:42.216240 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:42.216929 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:42.216951 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:42.219506 (+ 2555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:39:42.222836 (+ 3330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:42.222848 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:42.222869 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:42.223474 (+ 605us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:42.223855 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:42.223863 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:42.223869 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:42.224107 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:42.224590 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:39:42.224600 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:39:42.225065 (+ 465us) client_negotiation.cc:769] Sending connection context
0620 18:39:42.225419 (+ 354us) client_negotiation.cc:240] Negotiation successful
0620 18:39:42.226049 (+ 630us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":319,"threads_started":1}
I20260620 18:39:42.229043 6565 heartbeater.cc:347] Connected to a master server at 127.22.37.62:39147
I20260620 18:39:42.229404 6565 heartbeater.cc:464] Registering TS with master...
I20260620 18:39:42.230333 6565 heartbeater.cc:511] Master 127.22.37.62:39147 requested a full tablet report, sending...
I20260620 18:39:42.232010 6248 ts_manager.cc:194] Registered new tserver with Master: c995f8a8741a4540b650832e93ff4d36 (127.22.37.2:36829)
I20260620 18:39:42.233054 6248 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.37.2@KRBTEST.COM'} at 127.22.37.2:43903
W20260620 18:39:42.540377 6568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:42.540647 6568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:42.540741 6568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:42.548604 6568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:42.548713 6568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:42.548830 6568 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.37.3
I20260620 18:39:42.558737 6568 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:33907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.3.keytab
--principal=kudu/127.22.37.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.37.3:35831
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.37.3
--webserver_port=36447
--tserver_master_addrs=127.22.37.62:39147
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.37.3
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:39:42.560072 6568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:42.561347 6568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:39:42.574316 6574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:42.574394 6573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:42.574342 6576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:42.575285 6568 server_base.cc:1034] running on GCE node
I20260620 18:39:42.575939 6568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:42.576876 6568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:42.578075 6568 hybrid_clock.cc:648] HybridClock initialized: now 1781980782578008 us; error 56 us; skew 500 ppm
I20260620 18:39:42.578405 6568 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:39:42 dist-test-slave-5lgp krb5kdc[5664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980782, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:39:42.581483 6568 init.cc:377] Logged in from keytab as kudu/127.22.37.3@KRBTEST.COM (short username kudu)
I20260620 18:39:42.582962 6568 webserver.cc:460] Webserver started at http://127.22.37.3:36447/ using document root <none> and password file <none>
I20260620 18:39:42.583833 6568 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:42.583997 6568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:42.589274 6568 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 18:39:42.591868 6583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:42.593267 6568 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:39:42.593510 6568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-2/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-2/wal
uuid: "51e1565c5d3a45da9d90ee7d50e5184e"
format_stamp: "Formatted at 2026-06-20 18:39:19 on dist-test-slave-5lgp"
server_key: "27ff70d06d7fdb110dbc6b59aeaf247c"
server_key_iv: "3b466baf5f24c2a8f59829527b5af6de"
server_key_version: "encryptionkey@0"
I20260620 18:39:42.594539 6568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:42.607728 6568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:42.611948 6568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:42.612262 6568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:42.613296 6568 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:39:42.614899 6568 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:39:42.615037 6568 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:42.615168 6568 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:39:42.615265 6568 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:42.651671 6568 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.3:35831
I20260620 18:39:42.651733 6695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.3:35831 every 8 connection(s)
I20260620 18:39:42.653295 6568 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-2/data/info.pb
I20260620 18:39:42.661623 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 6568
Jun 20 18:39:42 dist-test-slave-5lgp krb5kdc[5664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980782, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.3@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM
I20260620 18:39:42.676060 6435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:42.656147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.3:46089 (local address 127.22.37.62:39147)
0620 18:39:42.656354 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:42.656360 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:42.657796 (+ 1436us) server_negotiation.cc:407] Connection header received
0620 18:39:42.659498 (+ 1702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:42.659504 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:42.659595 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:42.659705 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:42.661192 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:42.662184 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:42.664172 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:42.665117 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:42.665295 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:42.669028 (+ 3733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:42.669035 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:42.669040 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:42.669081 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:42.672649 (+ 3568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:42.673450 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:42.673456 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:42.673461 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:42.673598 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:42.674101 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:42.674106 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:42.674116 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:42.674493 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:42.674639 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:42.675322 (+ 683us) server_negotiation.cc:299] Negotiation successful
0620 18:39:42.675618 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 18:39:42.676728 6698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:42.656716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:39147 (local address 127.22.37.3:46089)
0620 18:39:42.657445 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:42.657501 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:42.659225 (+ 1724us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:42.659937 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:42.659945 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:42.660435 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:42.660998 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:42.661016 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:42.662374 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:42.662380 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:42.664012 (+ 1632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:42.664024 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:42.665340 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:42.665345 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:42.665743 (+ 398us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:42.666527 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:42.666551 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:42.668824 (+ 2273us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:39:42.672777 (+ 3953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:42.672803 (+ 26us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:42.672826 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:42.673304 (+ 478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:42.673731 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:42.673739 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:42.673744 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:42.673985 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:42.674633 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:39:42.674640 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:39:42.674954 (+ 314us) client_negotiation.cc:769] Sending connection context
0620 18:39:42.675187 (+ 233us) client_negotiation.cc:240] Negotiation successful
0620 18:39:42.675589 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":264,"threads_started":1}
I20260620 18:39:42.678369 6696 heartbeater.cc:347] Connected to a master server at 127.22.37.62:39147
I20260620 18:39:42.678697 6696 heartbeater.cc:464] Registering TS with master...
I20260620 18:39:42.679417 6696 heartbeater.cc:511] Master 127.22.37.62:39147 requested a full tablet report, sending...
I20260620 18:39:42.680596 6700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:42.662634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53814 (local address 127.22.37.62:39147)
0620 18:39:42.665795 (+ 3161us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:42.665801 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:42.665819 (+ 18us) server_negotiation.cc:407] Connection header received
0620 18:39:42.665871 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:42.665876 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:42.665963 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:42.666086 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:42.667798 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:42.668623 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:42.670527 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:42.671199 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:42.671417 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:42.674306 (+ 2889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:42.674313 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:42.674317 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:42.674345 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:42.677722 (+ 3377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:42.678563 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:42.678569 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:42.678572 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:42.678695 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:42.679112 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:42.679118 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:42.679121 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:42.679385 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:42.679649 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:42.679955 (+ 306us) server_negotiation.cc:299] Negotiation successful
0620 18:39:42.680183 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3052,"thread_start_us":2925,"threads_started":1}
I20260620 18:39:42.680984 6248 ts_manager.cc:194] Registered new tserver with Master: 51e1565c5d3a45da9d90ee7d50e5184e (127.22.37.3:35831)
I20260620 18:39:42.682010 6248 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.37.3@KRBTEST.COM'} at 127.22.37.3:46089
I20260620 18:39:42.687316 22676 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:39:42.693501 6700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:42.688186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53818 (local address 127.22.37.62:39147)
0620 18:39:42.688345 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:42.688349 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:42.688863 (+ 514us) server_negotiation.cc:407] Connection header received
0620 18:39:42.689042 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:42.689045 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:42.689165 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:42.689284 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:39:42.689796 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:42.690478 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:42.691863 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:42.692276 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:42.692363 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:42.692811 (+ 448us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:39:42.692988 (+ 177us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0620 18:39:42.693249 (+ 261us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53818: token expired
Metrics: {"server-negotiator.queue_time_us":54}
W20260620 18:39:42.693630 6700 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53818: token expired
W20260620 18:39:42.694200 6168 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.22.37.62:39147)
I20260620 18:39:42.705772 6700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:42.694693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53832 (local address 127.22.37.62:39147)
0620 18:39:42.694840 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:42.694844 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:42.694857 (+ 13us) server_negotiation.cc:407] Connection header received
0620 18:39:42.694893 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:42.694896 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:42.694942 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:42.695012 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:42.696273 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:42.696929 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:42.698219 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:42.698604 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:42.698724 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:42.701229 (+ 2505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:42.701234 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:42.701237 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:42.701265 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:42.703599 (+ 2334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:42.704198 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:42.704201 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:42.704204 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:42.704323 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:42.704619 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:42.704622 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:42.704624 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:42.704904 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:42.705079 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:42.705325 (+ 246us) server_negotiation.cc:299] Negotiation successful
0620 18:39:42.705532 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 18:39:42.709017 6247 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53832:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:39:42.716161 6247 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:39:42.739631 6708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:42.728318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51840 (local address 127.22.37.3:35831)
0620 18:39:42.729926 (+ 1608us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:42.729934 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:42.730751 (+ 817us) server_negotiation.cc:407] Connection header received
0620 18:39:42.731040 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:42.731048 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:42.731256 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:42.731429 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:39:42.732549 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:42.733692 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:42.736493 (+ 2801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:42.737989 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:42.738116 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:42.738361 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:42.739163 (+ 802us) server_negotiation.cc:299] Negotiation successful
0620 18:39:42.739329 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1443,"thread_start_us":1255,"threads_started":1}
I20260620 18:39:42.739691 6710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:42.728452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.1:42741 (local address 127.0.0.1:33280)
0620 18:39:42.730794 (+ 2342us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:42.730813 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:42.730907 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:42.731738 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:42.731741 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:42.731770 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:39:42.732273 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:42.732289 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:42.734186 (+ 1897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:42.734193 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:42.736619 (+ 2426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:42.736633 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:42.738707 (+ 2074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:42.738714 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:42.738904 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:42.738928 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 18:39:42.739004 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 18:39:42.739220 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2240,"mutex_wait_us":162,"spinlock_wait_cycles":896,"thread_start_us":2288,"threads_started":1}
I20260620 18:39:42.739675 6707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:42.727903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44896 (local address 127.22.37.2:36829)
0620 18:39:42.729335 (+ 1432us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:42.729348 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:42.730693 (+ 1345us) server_negotiation.cc:407] Connection header received
0620 18:39:42.730815 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:42.730823 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:42.731060 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:42.731250 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:39:42.732257 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:42.733919 (+ 1662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:42.737064 (+ 3145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:42.738648 (+ 1584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:42.738806 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:42.738981 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:42.739232 (+ 251us) server_negotiation.cc:299] Negotiation successful
0620 18:39:42.739388 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1183,"thread_start_us":1065,"threads_started":1}
I20260620 18:39:42.739964 6706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:42.727588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.2:36829 (local address 127.0.0.1:44896)
0620 18:39:42.730155 (+ 2567us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:42.730210 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:42.730451 (+ 241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:42.731496 (+ 1045us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:42.731503 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:42.731539 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:39:42.732039 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:42.732054 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:42.734098 (+ 2044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:42.734104 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:42.736881 (+ 2777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:42.736893 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:42.738870 (+ 1977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:42.738877 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:42.739038 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:42.739069 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 18:39:42.739183 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 18:39:42.739316 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2334,"mutex_wait_us":75,"thread_start_us":2164,"threads_started":1}
I20260620 18:39:42.740406 6711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:42.727865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.3:35831 (local address 127.0.0.1:51840)
0620 18:39:42.730611 (+ 2746us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:42.730630 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:42.730860 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:42.731731 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:42.731735 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:42.731768 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:39:42.732387 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:42.732396 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:42.733869 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:42.733875 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:42.736322 (+ 2447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:42.736339 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:42.738505 (+ 2166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:42.738511 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:42.738811 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:42.738876 (+ 65us) client_negotiation.cc:769] Sending connection context
0620 18:39:42.739004 (+ 128us) client_negotiation.cc:240] Negotiation successful
0620 18:39:42.740091 (+ 1087us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2588,"mutex_wait_us":1156,"spinlock_wait_cycles":2816,"thread_start_us":2685,"threads_started":1}
I20260620 18:39:42.741127 6709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:42.728916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33280 (local address 127.22.37.1:42741)
0620 18:39:42.730179 (+ 1263us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:42.730190 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:42.731017 (+ 827us) server_negotiation.cc:407] Connection header received
0620 18:39:42.731108 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:42.731114 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:42.731323 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:42.731475 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:39:42.732501 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:42.734029 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:42.736803 (+ 2774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:42.738244 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:42.738362 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:42.738558 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:42.740622 (+ 2064us) server_negotiation.cc:299] Negotiation successful
0620 18:39:42.740797 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1069,"spinlock_wait_cycles":19840,"thread_start_us":1033,"threads_started":1}
I20260620 18:39:42.743656 6500 tablet_service.cc:1467] Processing CreateTablet for tablet 388ffa0861f04da488834911de7bed8c (DEFAULT_TABLE table=test-table [id=713ea08501a44383be22fede040331f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:39:42.743659 6631 tablet_service.cc:1467] Processing CreateTablet for tablet 388ffa0861f04da488834911de7bed8c (DEFAULT_TABLE table=test-table [id=713ea08501a44383be22fede040331f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:39:42.744833 6500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 388ffa0861f04da488834911de7bed8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:39:42.744959 6631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 388ffa0861f04da488834911de7bed8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:39:42.747612 6368 tablet_service.cc:1467] Processing CreateTablet for tablet 388ffa0861f04da488834911de7bed8c (DEFAULT_TABLE table=test-table [id=713ea08501a44383be22fede040331f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:39:42.748869 6368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 388ffa0861f04da488834911de7bed8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:39:42.766134 6712 tablet_bootstrap.cc:492] T 388ffa0861f04da488834911de7bed8c P 51e1565c5d3a45da9d90ee7d50e5184e: Bootstrap starting.
I20260620 18:39:42.767385 6714 tablet_bootstrap.cc:492] T 388ffa0861f04da488834911de7bed8c P 129d88bd84a241a68291203014b805d0: Bootstrap starting.
I20260620 18:39:42.770860 6712 tablet_bootstrap.cc:654] T 388ffa0861f04da488834911de7bed8c P 51e1565c5d3a45da9d90ee7d50e5184e: Neither blocks nor log segments found. Creating new log.
I20260620 18:39:42.771375 6713 tablet_bootstrap.cc:492] T 388ffa0861f04da488834911de7bed8c P c995f8a8741a4540b650832e93ff4d36: Bootstrap starting.
I20260620 18:39:42.771936 6714 tablet_bootstrap.cc:654] T 388ffa0861f04da488834911de7bed8c P 129d88bd84a241a68291203014b805d0: Neither blocks nor log segments found. Creating new log.
I20260620 18:39:42.772635 6712 log.cc:824] T 388ffa0861f04da488834911de7bed8c P 51e1565c5d3a45da9d90ee7d50e5184e: Log is configured to *not* fsync() on all Append() calls
I20260620 18:39:42.773270 6714 log.cc:824] T 388ffa0861f04da488834911de7bed8c P 129d88bd84a241a68291203014b805d0: Log is configured to *not* fsync() on all Append() calls
I20260620 18:39:42.775175 6713 tablet_bootstrap.cc:654] T 388ffa0861f04da488834911de7bed8c P c995f8a8741a4540b650832e93ff4d36: Neither blocks nor log segments found. Creating new log.
I20260620 18:39:42.776108 6713 log.cc:824] T 388ffa0861f04da488834911de7bed8c P c995f8a8741a4540b650832e93ff4d36: Log is configured to *not* fsync() on all Append() calls
I20260620 18:39:42.776468 6712 tablet_bootstrap.cc:492] T 388ffa0861f04da488834911de7bed8c P 51e1565c5d3a45da9d90ee7d50e5184e: No bootstrap required, opened a new log
I20260620 18:39:42.776469 6714 tablet_bootstrap.cc:492] T 388ffa0861f04da488834911de7bed8c P 129d88bd84a241a68291203014b805d0: No bootstrap required, opened a new log
I20260620 18:39:42.776744 6714 ts_tablet_manager.cc:1397] T 388ffa0861f04da488834911de7bed8c P 129d88bd84a241a68291203014b805d0: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260620 18:39:42.776744 6712 ts_tablet_manager.cc:1397] T 388ffa0861f04da488834911de7bed8c P 51e1565c5d3a45da9d90ee7d50e5184e: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.006s
I20260620 18:39:42.778578 6713 tablet_bootstrap.cc:492] T 388ffa0861f04da488834911de7bed8c P c995f8a8741a4540b650832e93ff4d36: No bootstrap required, opened a new log
I20260620 18:39:42.778836 6713 ts_tablet_manager.cc:1397] T 388ffa0861f04da488834911de7bed8c P c995f8a8741a4540b650832e93ff4d36: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260620 18:39:42.786367 6713 raft_consensus.cc:348] T 388ffa0861f04da488834911de7bed8c P c995f8a8741a4540b650832e93ff4d36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c995f8a8741a4540b650832e93ff4d36" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 36829 } } peers { permanent_uuid: "51e1565c5d3a45da9d90ee7d50e5184e" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35831 } } peers { permanent_uuid: "129d88bd84a241a68291203014b805d0" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 42741 } }
I20260620 18:39:42.786728 6713 raft_consensus.cc:374] T 388ffa0861f04da488834911de7bed8c P c995f8a8741a4540b650832e93ff4d36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:39:42.786859 6713 raft_consensus.cc:729] T 388ffa0861f04da488834911de7bed8c P c995f8a8741a4540b650832e93ff4d36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c995f8a8741a4540b650832e93ff4d36, State: Initialized, Role: FOLLOWER
I20260620 18:39:42.787500 6713 consensus_queue.cc:260] T 388ffa0861f04da488834911de7bed8c P c995f8a8741a4540b650832e93ff4d36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c995f8a8741a4540b650832e93ff4d36" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 36829 } } peers { permanent_uuid: "51e1565c5d3a45da9d90ee7d50e5184e" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35831 } } peers { permanent_uuid: "129d88bd84a241a68291203014b805d0" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 42741 } }
I20260620 18:39:42.787449 6712 raft_consensus.cc:348] T 388ffa0861f04da488834911de7bed8c P 51e1565c5d3a45da9d90ee7d50e5184e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c995f8a8741a4540b650832e93ff4d36" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 36829 } } peers { permanent_uuid: "51e1565c5d3a45da9d90ee7d50e5184e" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35831 } } peers { permanent_uuid: "129d88bd84a241a68291203014b805d0" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 42741 } }
I20260620 18:39:42.787449 6714 raft_consensus.cc:348] T 388ffa0861f04da488834911de7bed8c P 129d88bd84a241a68291203014b805d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c995f8a8741a4540b650832e93ff4d36" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 36829 } } peers { permanent_uuid: "51e1565c5d3a45da9d90ee7d50e5184e" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35831 } } peers { permanent_uuid: "129d88bd84a241a68291203014b805d0" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 42741 } }
I20260620 18:39:42.787863 6714 raft_consensus.cc:374] T 388ffa0861f04da488834911de7bed8c P 129d88bd84a241a68291203014b805d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:39:42.787880 6712 raft_consensus.cc:374] T 388ffa0861f04da488834911de7bed8c P 51e1565c5d3a45da9d90ee7d50e5184e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:39:42.787963 6712 raft_consensus.cc:729] T 388ffa0861f04da488834911de7bed8c P 51e1565c5d3a45da9d90ee7d50e5184e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51e1565c5d3a45da9d90ee7d50e5184e, State: Initialized, Role: FOLLOWER
I20260620 18:39:42.788031 6714 raft_consensus.cc:729] T 388ffa0861f04da488834911de7bed8c P 129d88bd84a241a68291203014b805d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 129d88bd84a241a68291203014b805d0, State: Initialized, Role: FOLLOWER
I20260620 18:39:42.788558 6565 heartbeater.cc:503] Master 127.22.37.62:39147 was elected leader, sending a full tablet report...
I20260620 18:39:42.788749 6433 heartbeater.cc:503] Master 127.22.37.62:39147 was elected leader, sending a full tablet report...
I20260620 18:39:42.788678 6712 consensus_queue.cc:260] T 388ffa0861f04da488834911de7bed8c P 51e1565c5d3a45da9d90ee7d50e5184e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c995f8a8741a4540b650832e93ff4d36" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 36829 } } peers { permanent_uuid: "51e1565c5d3a45da9d90ee7d50e5184e" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35831 } } peers { permanent_uuid: "129d88bd84a241a68291203014b805d0" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 42741 } }
I20260620 18:39:42.789207 6713 ts_tablet_manager.cc:1428] T 388ffa0861f04da488834911de7bed8c P c995f8a8741a4540b650832e93ff4d36: Time spent starting tablet: real 0.010s user 0.011s sys 0.000s
I20260620 18:39:42.789633 6714 consensus_queue.cc:260] T 388ffa0861f04da488834911de7bed8c P 129d88bd84a241a68291203014b805d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c995f8a8741a4540b650832e93ff4d36" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 36829 } } peers { permanent_uuid: "51e1565c5d3a45da9d90ee7d50e5184e" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35831 } } peers { permanent_uuid: "129d88bd84a241a68291203014b805d0" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 42741 } }
I20260620 18:39:42.791118 6712 ts_tablet_manager.cc:1428] T 388ffa0861f04da488834911de7bed8c P 51e1565c5d3a45da9d90ee7d50e5184e: Time spent starting tablet: real 0.014s user 0.013s sys 0.001s
I20260620 18:39:42.791601 6696 heartbeater.cc:503] Master 127.22.37.62:39147 was elected leader, sending a full tablet report...
I20260620 18:39:42.796819 6714 ts_tablet_manager.cc:1428] T 388ffa0861f04da488834911de7bed8c P 129d88bd84a241a68291203014b805d0: Time spent starting tablet: real 0.020s user 0.013s sys 0.003s
W20260620 18:39:42.905274 6697 tablet.cc:2369] T 388ffa0861f04da488834911de7bed8c P 51e1565c5d3a45da9d90ee7d50e5184e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:39:42.905313 6720 raft_consensus.cc:482] T 388ffa0861f04da488834911de7bed8c P 129d88bd84a241a68291203014b805d0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:39:42.905650 6720 raft_consensus.cc:504] T 388ffa0861f04da488834911de7bed8c P 129d88bd84a241a68291203014b805d0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c995f8a8741a4540b650832e93ff4d36" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 36829 } } peers { permanent_uuid: "51e1565c5d3a45da9d90ee7d50e5184e" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35831 } } peers { permanent_uuid: "129d88bd84a241a68291203014b805d0" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 42741 } }
I20260620 18:39:42.907500 6720 leader_election.cc:288] T 388ffa0861f04da488834911de7bed8c P 129d88bd84a241a68291203014b805d0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c995f8a8741a4540b650832e93ff4d36 (127.22.37.2:36829), 51e1565c5d3a45da9d90ee7d50e5184e (127.22.37.3:35831)
I20260620 18:39:42.914283 6708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:42.908073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.1:39653 (local address 127.22.37.3:35831)
0620 18:39:42.908303 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:42.908314 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:42.908858 (+ 544us) server_negotiation.cc:407] Connection header received
0620 18:39:42.909031 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:42.909038 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:42.909147 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:42.909230 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:39:42.909863 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:42.910652 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:42.912480 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:42.913280 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:42.913429 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:42.913552 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:42.913906 (+ 354us) server_negotiation.cc:299] Negotiation successful
0620 18:39:42.914033 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 18:39:42.914338 6707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:42.907852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.1:45149 (local address 127.22.37.2:36829)
0620 18:39:42.908076 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:42.908084 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:42.909041 (+ 957us) server_negotiation.cc:407] Connection header received
0620 18:39:42.909180 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:42.909184 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:42.909286 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:42.909396 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:39:42.910112 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:42.910837 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:42.912534 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:42.913288 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:42.913482 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:42.913580 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:42.913953 (+ 373us) server_negotiation.cc:299] Negotiation successful
0620 18:39:42.914075 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 18:39:42.914426 6721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:42.907768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.3:35831 (local address 127.22.37.1:39653)
0620 18:39:42.908722 (+ 954us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:42.908741 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:42.908886 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:42.909279 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:42.909284 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:42.909310 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:39:42.909707 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:42.909721 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:42.910795 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:42.910800 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:42.912328 (+ 1528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:42.912340 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:42.913553 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:42.913560 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:42.913737 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:42.913776 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 18:39:42.913852 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 18:39:42.913994 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":806,"mutex_wait_us":15,"thread_start_us":578,"threads_started":1}
I20260620 18:39:42.914685 6722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:42.907892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.2:36829 (local address 127.22.37.1:45149)
0620 18:39:42.908879 (+ 987us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:42.908898 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:42.909055 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:42.909562 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:42.909566 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:42.909589 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:39:42.909959 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:42.909970 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:42.910985 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:42.910988 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:42.912360 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:42.912371 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:42.913608 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:42.913614 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:42.913773 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:42.913795 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 18:39:42.913903 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 18:39:42.914023 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":844,"mutex_wait_us":20,"thread_start_us":525,"threads_started":1}
I20260620 18:39:42.915616 6520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "388ffa0861f04da488834911de7bed8c" candidate_uuid: "129d88bd84a241a68291203014b805d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c995f8a8741a4540b650832e93ff4d36" is_pre_election: true
I20260620 18:39:42.915614 6651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "388ffa0861f04da488834911de7bed8c" candidate_uuid: "129d88bd84a241a68291203014b805d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51e1565c5d3a45da9d90ee7d50e5184e" is_pre_election: true
I20260620 18:39:42.916091 6651 raft_consensus.cc:2454] T 388ffa0861f04da488834911de7bed8c P 51e1565c5d3a45da9d90ee7d50e5184e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 129d88bd84a241a68291203014b805d0 in term 0.
I20260620 18:39:42.916074 6520 raft_consensus.cc:2454] T 388ffa0861f04da488834911de7bed8c P c995f8a8741a4540b650832e93ff4d36 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 129d88bd84a241a68291203014b805d0 in term 0.
I20260620 18:39:42.916728 6322 leader_election.cc:302] T 388ffa0861f04da488834911de7bed8c P 129d88bd84a241a68291203014b805d0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 129d88bd84a241a68291203014b805d0, 51e1565c5d3a45da9d90ee7d50e5184e, c995f8a8741a4540b650832e93ff4d36; no voters:
I20260620 18:39:42.917129 6720 raft_consensus.cc:2781] T 388ffa0861f04da488834911de7bed8c P 129d88bd84a241a68291203014b805d0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:39:42.917289 6720 raft_consensus.cc:482] T 388ffa0861f04da488834911de7bed8c P 129d88bd84a241a68291203014b805d0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:39:42.917441 6720 raft_consensus.cc:3037] T 388ffa0861f04da488834911de7bed8c P 129d88bd84a241a68291203014b805d0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:39:42.919611 6720 raft_consensus.cc:504] T 388ffa0861f04da488834911de7bed8c P 129d88bd84a241a68291203014b805d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c995f8a8741a4540b650832e93ff4d36" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 36829 } } peers { permanent_uuid: "51e1565c5d3a45da9d90ee7d50e5184e" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35831 } } peers { permanent_uuid: "129d88bd84a241a68291203014b805d0" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 42741 } }
I20260620 18:39:42.920261 6720 leader_election.cc:288] T 388ffa0861f04da488834911de7bed8c P 129d88bd84a241a68291203014b805d0 [CANDIDATE]: Term 1 election: Requested vote from peers c995f8a8741a4540b650832e93ff4d36 (127.22.37.2:36829), 51e1565c5d3a45da9d90ee7d50e5184e (127.22.37.3:35831)
I20260620 18:39:42.920662 6520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "388ffa0861f04da488834911de7bed8c" candidate_uuid: "129d88bd84a241a68291203014b805d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c995f8a8741a4540b650832e93ff4d36"
I20260620 18:39:42.920796 6651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "388ffa0861f04da488834911de7bed8c" candidate_uuid: "129d88bd84a241a68291203014b805d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51e1565c5d3a45da9d90ee7d50e5184e"
I20260620 18:39:42.920884 6520 raft_consensus.cc:3037] T 388ffa0861f04da488834911de7bed8c P c995f8a8741a4540b650832e93ff4d36 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:39:42.921015 6651 raft_consensus.cc:3037] T 388ffa0861f04da488834911de7bed8c P 51e1565c5d3a45da9d90ee7d50e5184e [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:39:42.924196 6651 raft_consensus.cc:2454] T 388ffa0861f04da488834911de7bed8c P 51e1565c5d3a45da9d90ee7d50e5184e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 129d88bd84a241a68291203014b805d0 in term 1.
I20260620 18:39:42.924209 6520 raft_consensus.cc:2454] T 388ffa0861f04da488834911de7bed8c P c995f8a8741a4540b650832e93ff4d36 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 129d88bd84a241a68291203014b805d0 in term 1.
I20260620 18:39:42.924597 6322 leader_election.cc:302] T 388ffa0861f04da488834911de7bed8c P 129d88bd84a241a68291203014b805d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 129d88bd84a241a68291203014b805d0, c995f8a8741a4540b650832e93ff4d36; no voters:
I20260620 18:39:42.924881 6720 raft_consensus.cc:2781] T 388ffa0861f04da488834911de7bed8c P 129d88bd84a241a68291203014b805d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:39:42.925325 6720 raft_consensus.cc:686] T 388ffa0861f04da488834911de7bed8c P 129d88bd84a241a68291203014b805d0 [term 1 LEADER]: Becoming Leader. State: Replica: 129d88bd84a241a68291203014b805d0, State: Running, Role: LEADER
I20260620 18:39:42.925834 6720 consensus_queue.cc:237] T 388ffa0861f04da488834911de7bed8c P 129d88bd84a241a68291203014b805d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c995f8a8741a4540b650832e93ff4d36" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 36829 } } peers { permanent_uuid: "51e1565c5d3a45da9d90ee7d50e5184e" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35831 } } peers { permanent_uuid: "129d88bd84a241a68291203014b805d0" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 42741 } }
I20260620 18:39:42.929994 6248 catalog_manager.cc:5495] T 388ffa0861f04da488834911de7bed8c P 129d88bd84a241a68291203014b805d0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 129d88bd84a241a68291203014b805d0 (127.22.37.1). New cstate: current_term: 1 leader_uuid: "129d88bd84a241a68291203014b805d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c995f8a8741a4540b650832e93ff4d36" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 36829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51e1565c5d3a45da9d90ee7d50e5184e" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "129d88bd84a241a68291203014b805d0" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 42741 } health_report { overall_health: HEALTHY } } }
W20260620 18:39:42.956867 6566 tablet.cc:2369] T 388ffa0861f04da488834911de7bed8c P c995f8a8741a4540b650832e93ff4d36: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:39:42.981297 6709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:42.975648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33282 (local address 127.22.37.1:42741)
0620 18:39:42.975866 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:42.975872 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:42.975988 (+ 116us) server_negotiation.cc:407] Connection header received
0620 18:39:42.976267 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:42.976272 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:42.976359 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:42.976510 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:39:42.976885 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:42.977713 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:42.979155 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:42.979582 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:42.979714 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:42.980225 (+ 511us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:39:42.980704 (+ 479us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:39:42.980873 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:42.980947 (+ 74us) server_negotiation.cc:299] Negotiation successful
0620 18:39:42.981026 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 18:39:42.989645 6520 raft_consensus.cc:1261] T 388ffa0861f04da488834911de7bed8c P c995f8a8741a4540b650832e93ff4d36 [term 1 FOLLOWER]: Refusing update from remote peer 129d88bd84a241a68291203014b805d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:39:42.989645 6651 raft_consensus.cc:1261] T 388ffa0861f04da488834911de7bed8c P 51e1565c5d3a45da9d90ee7d50e5184e [term 1 FOLLOWER]: Refusing update from remote peer 129d88bd84a241a68291203014b805d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:39:42.990701 6723 consensus_queue.cc:1035] T 388ffa0861f04da488834911de7bed8c P 129d88bd84a241a68291203014b805d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c995f8a8741a4540b650832e93ff4d36" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 36829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:39:42.991032 6720 consensus_queue.cc:1035] T 388ffa0861f04da488834911de7bed8c P 129d88bd84a241a68291203014b805d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51e1565c5d3a45da9d90ee7d50e5184e" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:39:43.003141 6726 mvcc.cc:205] Tried to move back new op lower bound from 7298993287115341824 to 7298993286867963904. Current Snapshot: MvccSnapshot[applied={T|T < 7298993287115341824}]
I20260620 18:39:43.016644 6248 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53832:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:39:43.016942 6248 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53832:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:39:43.020465 6248 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a593d764462a463d8334200095758268: Sending DeleteTablet for 3 replicas of tablet 388ffa0861f04da488834911de7bed8c
I20260620 18:39:43.021660 6631 tablet_service.cc:1514] Processing DeleteTablet for tablet 388ffa0861f04da488834911de7bed8c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:39:43 UTC) from {username='kudu', principal='kudu/127.22.37.62@KRBTEST.COM'} at 127.0.0.1:51840
I20260620 18:39:43.021780 6368 tablet_service.cc:1514] Processing DeleteTablet for tablet 388ffa0861f04da488834911de7bed8c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:39:43 UTC) from {username='kudu', principal='kudu/127.22.37.62@KRBTEST.COM'} at 127.0.0.1:33280
I20260620 18:39:43.022305 6500 tablet_service.cc:1514] Processing DeleteTablet for tablet 388ffa0861f04da488834911de7bed8c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:39:43 UTC) from {username='kudu', principal='kudu/127.22.37.62@KRBTEST.COM'} at 127.0.0.1:44896
I20260620 18:39:43.023139 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 6285
I20260620 18:39:43.054296 6709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:43.024320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33296 (local address 127.22.37.1:42741)
0620 18:39:43.024478 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:43.024484 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:43.024501 (+ 17us) server_negotiation.cc:407] Connection header received
0620 18:39:43.024555 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:43.024560 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:43.024673 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:43.024793 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:43.027029 (+ 2236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:43.028748 (+ 1719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:43.031580 (+ 2832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:43.032513 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:43.032661 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:43.041604 (+ 8943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:43.041625 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:43.041650 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:43.041706 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:43.047031 (+ 5325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:43.048224 (+ 1193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:43.048237 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:43.048244 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:43.048469 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:43.049998 (+ 1529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:43.050007 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:43.050012 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:43.050609 (+ 597us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:43.051044 (+ 435us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:43.052994 (+ 1950us) server_negotiation.cc:299] Negotiation successful
0620 18:39:43.053301 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 18:39:43.056206 6428 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:39:43.058609 6738 tablet_replica.cc:330] T 388ffa0861f04da488834911de7bed8c P c995f8a8741a4540b650832e93ff4d36: stopping tablet replica
I20260620 18:39:43.060130 6738 raft_consensus.cc:2229] T 388ffa0861f04da488834911de7bed8c P c995f8a8741a4540b650832e93ff4d36 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:39:43.061486 6738 raft_consensus.cc:2258] T 388ffa0861f04da488834911de7bed8c P c995f8a8741a4540b650832e93ff4d36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:39:43.062031 6737 tablet_replica.cc:330] T 388ffa0861f04da488834911de7bed8c P 51e1565c5d3a45da9d90ee7d50e5184e: stopping tablet replica
I20260620 18:39:43.062556 6737 raft_consensus.cc:2229] T 388ffa0861f04da488834911de7bed8c P 51e1565c5d3a45da9d90ee7d50e5184e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:39:43.063126 6737 raft_consensus.cc:2258] T 388ffa0861f04da488834911de7bed8c P 51e1565c5d3a45da9d90ee7d50e5184e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:39:43.064522 6738 ts_tablet_manager.cc:1905] T 388ffa0861f04da488834911de7bed8c P c995f8a8741a4540b650832e93ff4d36: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:39:43.064994 6737 ts_tablet_manager.cc:1905] T 388ffa0861f04da488834911de7bed8c P 51e1565c5d3a45da9d90ee7d50e5184e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:39:43.069274 6738 ts_tablet_manager.cc:1918] T 388ffa0861f04da488834911de7bed8c P c995f8a8741a4540b650832e93ff4d36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:39:43.069454 6738 log.cc:1196] T 388ffa0861f04da488834911de7bed8c P c995f8a8741a4540b650832e93ff4d36: Deleting WAL directory at /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-1/wal/wals/388ffa0861f04da488834911de7bed8c
I20260620 18:39:43.069909 6738 ts_tablet_manager.cc:1939] T 388ffa0861f04da488834911de7bed8c P c995f8a8741a4540b650832e93ff4d36: Deleting consensus metadata
I20260620 18:39:43.071580 6233 catalog_manager.cc:4841] TS c995f8a8741a4540b650832e93ff4d36 (127.22.37.2:36829): tablet 388ffa0861f04da488834911de7bed8c (table test-table [id=713ea08501a44383be22fede040331f9]) successfully deleted
I20260620 18:39:43.072310 6737 ts_tablet_manager.cc:1918] T 388ffa0861f04da488834911de7bed8c P 51e1565c5d3a45da9d90ee7d50e5184e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:39:43.072485 6737 log.cc:1196] T 388ffa0861f04da488834911de7bed8c P 51e1565c5d3a45da9d90ee7d50e5184e: Deleting WAL directory at /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-2/wal/wals/388ffa0861f04da488834911de7bed8c
I20260620 18:39:43.073017 6737 ts_tablet_manager.cc:1939] T 388ffa0861f04da488834911de7bed8c P 51e1565c5d3a45da9d90ee7d50e5184e: Deleting consensus metadata
I20260620 18:39:43.074920 6232 catalog_manager.cc:4841] TS 51e1565c5d3a45da9d90ee7d50e5184e (127.22.37.3:35831): tablet 388ffa0861f04da488834911de7bed8c (table test-table [id=713ea08501a44383be22fede040331f9]) successfully deleted
I20260620 18:39:43.165726 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 6285
I20260620 18:39:43.185545 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 6437
W20260620 18:39:43.185650 6234 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.37.1:42741 (error 108)
I20260620 18:39:43.187403 6710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:43.186934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.1:42741 (local address 127.0.0.1:33306)
0620 18:39:43.187123 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:43.187250 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.1:42741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260620 18:39:43.187646 6234 catalog_manager.cc:4584] TS 129d88bd84a241a68291203014b805d0 (127.22.37.1:42741): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 388ffa0861f04da488834911de7bed8c: Network error: Client connection negotiation failed: client connection to 127.22.37.1:42741: connect: Connection refused (error 111)
I20260620 18:39:43.199041 6707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:43.186388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44904 (local address 127.22.37.2:36829)
0620 18:39:43.186577 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:43.186583 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:43.186663 (+ 80us) server_negotiation.cc:407] Connection header received
0620 18:39:43.186887 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:43.186892 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:43.187002 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:43.187110 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:43.188575 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:43.189320 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:43.190769 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:43.191150 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:43.191244 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:43.193798 (+ 2554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:43.193811 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:43.193826 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:43.193859 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:43.196482 (+ 2623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:43.197083 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:43.197090 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:43.197092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:43.197227 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:43.197653 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:43.197656 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:43.197658 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:43.198036 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:43.198312 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:43.198505 (+ 193us) server_negotiation.cc:299] Negotiation successful
0620 18:39:43.198667 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 18:39:43.199772 6560 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:39:43.241257 6710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:43.240789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.1:42741 (local address 127.0.0.1:33310)
0620 18:39:43.240999 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:43.241102 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.1:42741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260620 18:39:43.242339 6710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:43.241988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.1:42741 (local address 127.0.0.1:33324)
0620 18:39:43.242138 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:43.242238 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.1:42741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 18:39:43.302979 6710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:43.302456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.1:42741 (local address 127.0.0.1:33326)
0620 18:39:43.302685 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:43.302786 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.1:42741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260620 18:39:43.304059 6710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:43.303684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.1:42741 (local address 127.0.0.1:33336)
0620 18:39:43.303837 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:43.303922 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.1:42741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 18:39:43.305807 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 6437
I20260620 18:39:43.323302 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 6568
I20260620 18:39:43.337576 6708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:43.324076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51842 (local address 127.22.37.3:35831)
0620 18:39:43.324260 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:43.324267 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:43.324377 (+ 110us) server_negotiation.cc:407] Connection header received
0620 18:39:43.324616 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:43.324620 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:43.324695 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:43.324801 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:43.326176 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:43.326905 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:43.328247 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:43.328630 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:43.328764 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:43.331517 (+ 2753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:43.331533 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:43.331550 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:43.331582 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:43.334582 (+ 3000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:43.335275 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:43.335285 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:43.335289 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:43.335501 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:43.335885 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:43.335892 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:43.335895 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:43.336256 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:43.336566 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:43.336878 (+ 312us) server_negotiation.cc:299] Negotiation successful
0620 18:39:43.337117 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 18:39:43.338232 6691 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:39:43.384696 6710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:43.384201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.1:42741 (local address 127.0.0.1:33352)
0620 18:39:43.384417 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:43.384504 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.1:42741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260620 18:39:43.385850 6710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:43.385470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.1:42741 (local address 127.0.0.1:33360)
0620 18:39:43.385663 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:43.385748 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.1:42741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260620 18:39:43.428897 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 6568
I20260620 18:39:43.445678 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 6215
I20260620 18:39:43.446861 6278 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:39:43.563930 6710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:43.563232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.1:42741 (local address 127.0.0.1:33374)
0620 18:39:43.563686 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:43.563773 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.1:42741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337}
I20260620 18:39:43.563930 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 6215
I20260620 18:40:03.586083 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.62:39147
--webserver_interface=127.22.37.62
--webserver_port=34685
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab
--principal=kudu/127.22.37.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:33907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.37.62:39147
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:40:03.918861 6744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:40:03.919265 6744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:40:03.919448 6744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:40:03.926796 6744 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 18:40:03.926947 6744 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 18:40:03.927021 6744 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 18:40:03.927099 6744 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:40:03.927179 6744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:40:03.927281 6744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:40:03.927350 6744 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:40:03.927428 6744 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:40:03.937445 6744 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:33907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.22.37.62:39147
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab
--principal=kudu/127.22.37.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.62:39147
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.37.62
--webserver_port=34685
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:40:03.938846 6744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:40:03.940289 6744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:40:03.952427 6749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:40:03.952468 6752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:40:03.952374 6750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:40:03.953311 6744 server_base.cc:1034] running on GCE node
I20260620 18:40:03.954216 6744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:40:03.955653 6744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:40:03.956887 6744 hybrid_clock.cc:648] HybridClock initialized: now 1781980803956826 us; error 35 us; skew 500 ppm
I20260620 18:40:03.957260 6744 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:40:03 dist-test-slave-5lgp krb5kdc[5664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980803, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:40:03.960525 6744 init.cc:377] Logged in from keytab as kudu/127.22.37.62@KRBTEST.COM (short username kudu)
I20260620 18:40:03.962118 6744 webserver.cc:460] Webserver started at http://127.22.37.62:34685/ using document root <none> and password file <none>
I20260620 18:40:03.962853 6744 fs_manager.cc:359] Metadata directory not provided
I20260620 18:40:03.962991 6744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:40:03.968763 6744 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 18:40:03.971839 6759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:40:03.973352 6744 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 18:40:03.973615 6744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/master-0/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/master-0/wal
uuid: "a593d764462a463d8334200095758268"
format_stamp: "Formatted at 2026-06-20 18:39:17 on dist-test-slave-5lgp"
server_key: "9262f4c61f5b39011a2d4c9108445344"
server_key_iv: "f62851751da673374e04ecb83bbba083"
server_key_version: "encryptionkey@0"
I20260620 18:40:03.974666 6744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:40:03.994141 6744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:40:03.998350 6744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:40:03.998649 6744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:40:04.017936 6744 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.62:39147
I20260620 18:40:04.017966 6810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.62:39147 every 8 connection(s)
I20260620 18:40:04.019707 6744 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
I20260620 18:40:04.020299 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 6744
I20260620 18:40:04.021353 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.37.1:42741
--local_ip_for_outbound_sockets=127.22.37.1
--tserver_master_addrs=127.22.37.62:39147
--webserver_port=46361
--webserver_interface=127.22.37.1
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.1.keytab
--principal=kudu/127.22.37.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:33907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:40:04.027881 6811 sys_catalog.cc:263] Verifying existing consensus state
I20260620 18:40:04.031786 6811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a593d764462a463d8334200095758268: Bootstrap starting.
I20260620 18:40:04.053835 6811 log.cc:824] T 00000000000000000000000000000000 P a593d764462a463d8334200095758268: Log is configured to *not* fsync() on all Append() calls
I20260620 18:40:04.070242 6811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a593d764462a463d8334200095758268: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 18:40:04.070833 6811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a593d764462a463d8334200095758268: Bootstrap complete.
I20260620 18:40:04.080336 6811 raft_consensus.cc:348] T 00000000000000000000000000000000 P a593d764462a463d8334200095758268 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a593d764462a463d8334200095758268" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 39147 } }
I20260620 18:40:04.081337 6811 raft_consensus.cc:729] T 00000000000000000000000000000000 P a593d764462a463d8334200095758268 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: a593d764462a463d8334200095758268, State: Initialized, Role: FOLLOWER
I20260620 18:40:04.082063 6811 consensus_queue.cc:260] T 00000000000000000000000000000000 P a593d764462a463d8334200095758268 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a593d764462a463d8334200095758268" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 39147 } }
I20260620 18:40:04.082321 6811 raft_consensus.cc:388] T 00000000000000000000000000000000 P a593d764462a463d8334200095758268 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:40:04.082446 6811 raft_consensus.cc:482] T 00000000000000000000000000000000 P a593d764462a463d8334200095758268 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:40:04.082638 6811 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a593d764462a463d8334200095758268 [term 2 FOLLOWER]: Advancing to term 3
I20260620 18:40:04.086560 6811 raft_consensus.cc:504] T 00000000000000000000000000000000 P a593d764462a463d8334200095758268 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a593d764462a463d8334200095758268" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 39147 } }
I20260620 18:40:04.087072 6811 leader_election.cc:302] T 00000000000000000000000000000000 P a593d764462a463d8334200095758268 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a593d764462a463d8334200095758268; no voters:
I20260620 18:40:04.087775 6811 leader_election.cc:288] T 00000000000000000000000000000000 P a593d764462a463d8334200095758268 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 18:40:04.087934 6815 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a593d764462a463d8334200095758268 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 18:40:04.089116 6815 raft_consensus.cc:686] T 00000000000000000000000000000000 P a593d764462a463d8334200095758268 [term 3 LEADER]: Becoming Leader. State: Replica: a593d764462a463d8334200095758268, State: Running, Role: LEADER
I20260620 18:40:04.089771 6815 consensus_queue.cc:237] T 00000000000000000000000000000000 P a593d764462a463d8334200095758268 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a593d764462a463d8334200095758268" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 39147 } }
I20260620 18:40:04.090058 6811 sys_catalog.cc:564] T 00000000000000000000000000000000 P a593d764462a463d8334200095758268 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:40:04.092236 6817 sys_catalog.cc:455] T 00000000000000000000000000000000 P a593d764462a463d8334200095758268 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a593d764462a463d8334200095758268. Latest consensus state: current_term: 3 leader_uuid: "a593d764462a463d8334200095758268" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a593d764462a463d8334200095758268" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 39147 } } }
I20260620 18:40:04.092887 6817 sys_catalog.cc:458] T 00000000000000000000000000000000 P a593d764462a463d8334200095758268 [sys.catalog]: This master's current role is: LEADER
I20260620 18:40:04.092953 6816 sys_catalog.cc:455] T 00000000000000000000000000000000 P a593d764462a463d8334200095758268 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "a593d764462a463d8334200095758268" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a593d764462a463d8334200095758268" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 39147 } } }
I20260620 18:40:04.094799 6816 sys_catalog.cc:458] T 00000000000000000000000000000000 P a593d764462a463d8334200095758268 [sys.catalog]: This master's current role is: LEADER
I20260620 18:40:04.097590 6823 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:40:04.107362 6823 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:40:04.108992 6823 catalog_manager.cc:1259] Loaded cluster ID: a5cfbde2c9ad4e62b179da5e8af50d12
I20260620 18:40:04.109175 6823 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:40:04.113461 6823 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:40:04.115227 6823 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a593d764462a463d8334200095758268: Loaded TSK: 0
I20260620 18:40:04.115466 6823 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a593d764462a463d8334200095758268: Loaded TSK: 1
F20260620 18:40:04.116685 6823 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f1c2ffc49ad google::LogMessage::Fail() at ??:0
@ 0x7f1c2ffc788a google::LogMessage::SendToLog() at ??:0
@ 0x7f1c2ffc44de google::LogMessage::Flush() at ??:0
@ 0x7f1c2ffc5a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f1c33d287d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f1c33d1773e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f1c459f31c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f1c459f2783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f1c45a7e2a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f1c45a7e0c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f1c45ac85e9 std::function<>::operator()() at ??:0
@ 0x7f1c459f6cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f1c459f4c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f1c45a7c610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f1c45a7c439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f1c32c10345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f1c32c1391c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f1c32c13799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f1c32beeeaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f1c35ec86db start_thread at ??:0
@ 0x7f1c2ee5e61f clone at ??:0
W20260620 18:40:04.446034 6813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:40:04.446360 6813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:40:04.446519 6813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:40:04.454290 6813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:40:04.454417 6813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:40:04.454627 6813 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.37.1
I20260620 18:40:04.465140 6813 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:33907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.1.keytab
--principal=kudu/127.22.37.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.37.1:42741
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.37.1
--webserver_port=46361
--tserver_master_addrs=127.22.37.62:39147
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.37.1
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:40:04.466629 6813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:40:04.468132 6813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:40:04.482144 6858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:40:04.482116 6861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:40:04.482064 6859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:40:04.483413 6813 server_base.cc:1034] running on GCE node
I20260620 18:40:04.484136 6813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:40:04.485191 6813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:40:04.486464 6813 hybrid_clock.cc:648] HybridClock initialized: now 1781980804486397 us; error 57 us; skew 500 ppm
I20260620 18:40:04.486850 6813 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:40:04 dist-test-slave-5lgp krb5kdc[5664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980804, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:40:04.496429 6813 init.cc:377] Logged in from keytab as kudu/127.22.37.1@KRBTEST.COM (short username kudu)
I20260620 18:40:04.498219 6813 webserver.cc:460] Webserver started at http://127.22.37.1:46361/ using document root <none> and password file <none>
I20260620 18:40:04.499003 6813 fs_manager.cc:359] Metadata directory not provided
I20260620 18:40:04.499162 6813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:40:04.505025 6813 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 18:40:04.508296 6868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:40:04.509868 6813 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:40:04.510015 6813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-0/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-0/wal
uuid: "129d88bd84a241a68291203014b805d0"
format_stamp: "Formatted at 2026-06-20 18:39:18 on dist-test-slave-5lgp"
server_key: "5fa8a0c42fb597c961965ef437c4e92a"
server_key_iv: "a199ce8a6a32e88fd610debef841e3df"
server_key_version: "encryptionkey@0"
I20260620 18:40:04.510990 6813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:40:04.545086 6813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:40:04.549755 6813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:40:04.550014 6813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:40:04.551005 6813 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:40:04.552814 6875 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260620 18:40:04.559186 6813 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 18:40:04.559378 6813 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.001s sys 0.000s
I20260620 18:40:04.559547 6813 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 18:40:04.562933 6813 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 18:40:04.563033 6813 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s user 0.002s sys 0.000s
I20260620 18:40:04.563475 6875 tablet_bootstrap.cc:492] T 388ffa0861f04da488834911de7bed8c P 129d88bd84a241a68291203014b805d0: Bootstrap starting.
I20260620 18:40:04.592588 6875 log.cc:824] T 388ffa0861f04da488834911de7bed8c P 129d88bd84a241a68291203014b805d0: Log is configured to *not* fsync() on all Append() calls
I20260620 18:40:04.603820 6813 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.1:42741
I20260620 18:40:04.603858 6982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.1:42741 every 8 connection(s)
I20260620 18:40:04.605176 6813 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-0/data/info.pb
I20260620 18:40:04.607896 6875 tablet_bootstrap.cc:492] T 388ffa0861f04da488834911de7bed8c P 129d88bd84a241a68291203014b805d0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 18:40:04.608436 6875 tablet_bootstrap.cc:492] T 388ffa0861f04da488834911de7bed8c P 129d88bd84a241a68291203014b805d0: Bootstrap complete.
I20260620 18:40:04.609840 6875 ts_tablet_manager.cc:1397] T 388ffa0861f04da488834911de7bed8c P 129d88bd84a241a68291203014b805d0: Time spent bootstrapping tablet: real 0.047s user 0.019s sys 0.026s
I20260620 18:40:04.610762 6985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:04.608372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:39147 (local address 127.22.37.1:35237)
0620 18:40:04.609361 (+ 989us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:04.609699 (+ 338us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:39147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":702,"thread_start_us":462,"threads_started":1}
I20260620 18:40:04.611021 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 6813
I20260620 18:40:04.611824 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.37.2:36829
--local_ip_for_outbound_sockets=127.22.37.2
--tserver_master_addrs=127.22.37.62:39147
--webserver_port=44147
--webserver_interface=127.22.37.2
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.2.keytab
--principal=kudu/127.22.37.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:33907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:40:04.613821 6983 heartbeater.cc:644] Failed to heartbeat to 127.22.37.62:39147 (0 consecutive failures): Network error: Failed to ping master at 127.22.37.62:39147: Client connection negotiation failed: client connection to 127.22.37.62:39147: connect: Connection refused (error 111)
I20260620 18:40:04.615023 6985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:04.614490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:39147 (local address 127.22.37.1:42953)
0620 18:40:04.614764 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:04.614850 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:39147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260620 18:40:04.616104 6985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:04.615658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:39147 (local address 127.22.37.1:50517)
0620 18:40:04.615855 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:04.615936 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:39147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
W20260620 18:40:04.616357 6983 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 18:40:04.621507 6875 raft_consensus.cc:348] T 388ffa0861f04da488834911de7bed8c P 129d88bd84a241a68291203014b805d0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c995f8a8741a4540b650832e93ff4d36" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 36829 } } peers { permanent_uuid: "51e1565c5d3a45da9d90ee7d50e5184e" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35831 } } peers { permanent_uuid: "129d88bd84a241a68291203014b805d0" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 42741 } }
I20260620 18:40:04.623121 6875 raft_consensus.cc:729] T 388ffa0861f04da488834911de7bed8c P 129d88bd84a241a68291203014b805d0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 129d88bd84a241a68291203014b805d0, State: Initialized, Role: FOLLOWER
I20260620 18:40:04.623900 6875 consensus_queue.cc:260] T 388ffa0861f04da488834911de7bed8c P 129d88bd84a241a68291203014b805d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c995f8a8741a4540b650832e93ff4d36" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 36829 } } peers { permanent_uuid: "51e1565c5d3a45da9d90ee7d50e5184e" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35831 } } peers { permanent_uuid: "129d88bd84a241a68291203014b805d0" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 42741 } }
I20260620 18:40:04.625670 6985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:04.625267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:39147 (local address 127.22.37.1:52317)
0620 18:40:04.625448 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:04.625540 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:39147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 18:40:04.625883 6875 ts_tablet_manager.cc:1428] T 388ffa0861f04da488834911de7bed8c P 129d88bd84a241a68291203014b805d0: Time spent starting tablet: real 0.016s user 0.014s sys 0.002s
I20260620 18:40:04.626817 6985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:04.626482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:39147 (local address 127.22.37.1:58689)
0620 18:40:04.626623 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:04.626697 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:39147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 18:40:04.948635 6986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:40:04.948915 6986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:40:04.949070 6986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:40:04.956503 6986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:40:04.956681 6986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:40:04.956875 6986 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.37.2
I20260620 18:40:04.966998 6986 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:33907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.2.keytab
--principal=kudu/127.22.37.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.37.2:36829
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.37.2
--webserver_port=44147
--tserver_master_addrs=127.22.37.62:39147
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.37.2
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:40:04.968427 6986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:40:04.969801 6986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:40:04.982934 6992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:40:04.982911 6993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:40:04.982910 6995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:40:04.983865 6986 server_base.cc:1034] running on GCE node
I20260620 18:40:04.984496 6986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:40:04.985513 6986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:40:04.986747 6986 hybrid_clock.cc:648] HybridClock initialized: now 1781980804986687 us; error 42 us; skew 500 ppm
I20260620 18:40:04.987072 6986 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:40:04 dist-test-slave-5lgp krb5kdc[5664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980804, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:40:04.990309 6986 init.cc:377] Logged in from keytab as kudu/127.22.37.2@KRBTEST.COM (short username kudu)
I20260620 18:40:04.991811 6986 webserver.cc:460] Webserver started at http://127.22.37.2:44147/ using document root <none> and password file <none>
I20260620 18:40:04.992535 6986 fs_manager.cc:359] Metadata directory not provided
I20260620 18:40:04.992690 6986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:40:04.998425 6986 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 18:40:05.001236 7002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:40:05.002702 6986 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 18:40:05.002918 6986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-1/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-1/wal
uuid: "c995f8a8741a4540b650832e93ff4d36"
format_stamp: "Formatted at 2026-06-20 18:39:19 on dist-test-slave-5lgp"
server_key: "1125f2023af9f5c615e75957e6bcb2aa"
server_key_iv: "e593df8bfc77c25c10e5285125c17478"
server_key_version: "encryptionkey@0"
I20260620 18:40:05.004022 6986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:40:05.017323 6986 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:40:05.021860 6986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:40:05.022161 6986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:40:05.023221 6986 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:40:05.024614 6986 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:40:05.024762 6986 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:40:05.024895 6986 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:40:05.024986 6986 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:40:05.059092 6986 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.2:36829
I20260620 18:40:05.059183 7114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.2:36829 every 8 connection(s)
I20260620 18:40:05.060600 6986 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-1/data/info.pb
I20260620 18:40:05.065687 7117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:05.063804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:39147 (local address 127.22.37.2:41373)
0620 18:40:05.064517 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:05.064714 (+ 197us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:39147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":286,"threads_started":1}
W20260620 18:40:05.066437 7115 heartbeater.cc:644] Failed to heartbeat to 127.22.37.62:39147 (0 consecutive failures): Network error: Failed to ping master at 127.22.37.62:39147: Client connection negotiation failed: client connection to 127.22.37.62:39147: connect: Connection refused (error 111)
I20260620 18:40:05.067238 7117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:05.066943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:39147 (local address 127.22.37.2:49245)
0620 18:40:05.067072 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:05.067138 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:39147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 18:40:05.068048 7117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:05.067779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:39147 (local address 127.22.37.2:48539)
0620 18:40:05.067889 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:05.067954 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:39147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 18:40:05.068324 7115 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 18:40:05.070031 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 6986
I20260620 18:40:05.070705 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.37.3:35831
--local_ip_for_outbound_sockets=127.22.37.3
--tserver_master_addrs=127.22.37.62:39147
--webserver_port=36447
--webserver_interface=127.22.37.3
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.3.keytab
--principal=kudu/127.22.37.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:33907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:40:05.400314 7118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:40:05.400593 7118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:40:05.400795 7118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:40:05.407998 7118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:40:05.408169 7118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:40:05.408366 7118 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.37.3
I20260620 18:40:05.418286 7118 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:33907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.3.keytab
--principal=kudu/127.22.37.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.37.3:35831
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.37.3
--webserver_port=36447
--tserver_master_addrs=127.22.37.62:39147
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.37.3
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:40:05.419696 7118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:40:05.421077 7118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:40:05.434136 7123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:40:05.434161 7124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:40:05.434069 7126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:40:05.435009 7118 server_base.cc:1034] running on GCE node
I20260620 18:40:05.435752 7118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:40:05.436741 7118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:40:05.437990 7118 hybrid_clock.cc:648] HybridClock initialized: now 1781980805437937 us; error 45 us; skew 500 ppm
I20260620 18:40:05.438362 7118 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:40:05 dist-test-slave-5lgp krb5kdc[5664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980805, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:40:05.441977 7118 init.cc:377] Logged in from keytab as kudu/127.22.37.3@KRBTEST.COM (short username kudu)
I20260620 18:40:05.443480 7118 webserver.cc:460] Webserver started at http://127.22.37.3:36447/ using document root <none> and password file <none>
I20260620 18:40:05.444202 7118 fs_manager.cc:359] Metadata directory not provided
I20260620 18:40:05.444388 7118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:40:05.450223 7118 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 18:40:05.453102 7133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:40:05.454614 7118 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 18:40:05.454810 7118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-2/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-2/wal
uuid: "51e1565c5d3a45da9d90ee7d50e5184e"
format_stamp: "Formatted at 2026-06-20 18:39:19 on dist-test-slave-5lgp"
server_key: "27ff70d06d7fdb110dbc6b59aeaf247c"
server_key_iv: "3b466baf5f24c2a8f59829527b5af6de"
server_key_version: "encryptionkey@0"
I20260620 18:40:05.455814 7118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:40:05.477540 7118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:40:05.481655 7118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:40:05.481942 7118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:40:05.482933 7118 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:40:05.484308 7118 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:40:05.484437 7118 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:40:05.484580 7118 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:40:05.484671 7118 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:40:05.517575 7118 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.3:35831
I20260620 18:40:05.517661 7245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.3:35831 every 8 connection(s)
I20260620 18:40:05.519130 7118 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980527431848-22676-0/minicluster-data/ts-2/data/info.pb
I20260620 18:40:05.524067 7248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:05.522222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:39147 (local address 127.22.37.3:57305)
0620 18:40:05.522931 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:05.523168 (+ 237us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:39147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":260,"threads_started":1}
I20260620 18:40:05.524363 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 7118
W20260620 18:40:05.524814 7246 heartbeater.cc:644] Failed to heartbeat to 127.22.37.62:39147 (0 consecutive failures): Network error: Failed to ping master at 127.22.37.62:39147: Client connection negotiation failed: client connection to 127.22.37.62:39147: connect: Connection refused (error 111)
I20260620 18:40:05.525760 7248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:05.525325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:39147 (local address 127.22.37.3:54685)
0620 18:40:05.525518 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:05.525625 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:39147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260620 18:40:05.526634 7248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:05.526356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:39147 (local address 127.22.37.3:48127)
0620 18:40:05.526469 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:05.526537 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:39147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 18:40:05.526895 7246 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 18:40:05.628677 7250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:05.627825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:39147 (local address 127.22.37.1:34075)
0620 18:40:05.628374 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:05.628453 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:39147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":303,"threads_started":1}
I20260620 18:40:06.069770 7251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:06.068933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:39147 (local address 127.22.37.2:55035)
0620 18:40:06.069451 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:06.069531 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:39147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":221,"threads_started":1}
I20260620 18:40:06.458523 7252 raft_consensus.cc:482] T 388ffa0861f04da488834911de7bed8c P 129d88bd84a241a68291203014b805d0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:40:06.458870 7252 raft_consensus.cc:504] T 388ffa0861f04da488834911de7bed8c P 129d88bd84a241a68291203014b805d0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c995f8a8741a4540b650832e93ff4d36" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 36829 } } peers { permanent_uuid: "51e1565c5d3a45da9d90ee7d50e5184e" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35831 } } peers { permanent_uuid: "129d88bd84a241a68291203014b805d0" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 42741 } }
I20260620 18:40:06.460690 7252 leader_election.cc:288] T 388ffa0861f04da488834911de7bed8c P 129d88bd84a241a68291203014b805d0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c995f8a8741a4540b650832e93ff4d36 (127.22.37.2:36829), 51e1565c5d3a45da9d90ee7d50e5184e (127.22.37.3:35831)
Jun 20 18:40:06 dist-test-slave-5lgp krb5kdc[5664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980804, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.1@KRBTEST.COM for kudu/127.22.37.3@KRBTEST.COM
Jun 20 18:40:06 dist-test-slave-5lgp krb5kdc[5664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980804, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.1@KRBTEST.COM for kudu/127.22.37.2@KRBTEST.COM
I20260620 18:40:06.485311 7255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:06.461345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.1:40831 (local address 127.22.37.3:35831)
0620 18:40:06.463779 (+ 2434us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:06.463797 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:06.463834 (+ 37us) server_negotiation.cc:407] Connection header received
0620 18:40:06.465414 (+ 1580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:06.465432 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:06.465832 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:06.466180 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:40:06.468173 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:06.469414 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:06.472233 (+ 2819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:06.472869 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:06.473001 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:06.477142 (+ 4141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:40:06.477150 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:40:06.477183 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:40:06.477224 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:40:06.481045 (+ 3821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:40:06.481905 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:40:06.481911 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:40:06.481918 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:40:06.482071 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:40:06.482544 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:40:06.482548 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:40:06.482550 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:40:06.483262 (+ 712us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:40:06.483448 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:06.484478 (+ 1030us) server_negotiation.cc:299] Negotiation successful
0620 18:40:06.484890 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2270,"thread_start_us":2133,"threads_started":1}
I20260620 18:40:06.485306 7256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:06.461521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.1:56445 (local address 127.22.37.2:36829)
0620 18:40:06.463286 (+ 1765us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:06.463303 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:06.463757 (+ 454us) server_negotiation.cc:407] Connection header received
0620 18:40:06.465789 (+ 2032us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:06.465798 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:06.466122 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:06.466437 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:40:06.468158 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:06.469343 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:06.473193 (+ 3850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:06.474080 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:06.474267 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:06.477733 (+ 3466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:40:06.477744 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:40:06.477762 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:40:06.477822 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:40:06.481511 (+ 3689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:40:06.482243 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:40:06.482249 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:40:06.482254 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:40:06.482392 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:40:06.482746 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:40:06.482751 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:40:06.482754 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:40:06.483322 (+ 568us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:40:06.483473 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:06.484478 (+ 1005us) server_negotiation.cc:299] Negotiation successful
0620 18:40:06.484890 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1539,"thread_start_us":1331,"threads_started":1}
I20260620 18:40:06.485292 7254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:06.460987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.2:36829 (local address 127.22.37.1:56445)
0620 18:40:06.463570 (+ 2583us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:06.463591 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 18:40:06.464098 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:40:06.466398 (+ 2300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:40:06.466402 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:40:06.467213 (+ 811us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:40:06.467958 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:06.467979 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:06.469599 (+ 1620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:06.469605 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:06.472092 (+ 2487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:06.472106 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:06.474286 (+ 2180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:06.474292 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:06.474489 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:06.475483 (+ 994us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:40:06.475513 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:40:06.477473 (+ 1960us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:40:06.481647 (+ 4174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:40:06.481655 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:40:06.481660 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:40:06.482091 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:40:06.482511 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:40:06.482517 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:40:06.482520 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:40:06.482640 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:40:06.483439 (+ 799us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:40:06.483442 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:40:06.483893 (+ 451us) client_negotiation.cc:769] Sending connection context
0620 18:40:06.484226 (+ 333us) client_negotiation.cc:240] Negotiation successful
0620 18:40:06.484779 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2434,"mutex_wait_us":490,"spinlock_wait_cycles":5376,"thread_start_us":2069,"threads_started":1}
I20260620 18:40:06.485316 7253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:06.460982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.3:35831 (local address 127.22.37.1:40831)
0620 18:40:06.461584 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:06.461642 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 18:40:06.462325 (+ 683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:40:06.466320 (+ 3995us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:40:06.466331 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:40:06.467261 (+ 930us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:40:06.467958 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:06.467979 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:06.469748 (+ 1769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:06.469752 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:06.472091 (+ 2339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:06.472106 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:06.473024 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:06.473028 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:06.473208 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:06.474005 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:40:06.474029 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:40:06.476836 (+ 2807us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:40:06.481188 (+ 4352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:40:06.481195 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:40:06.481207 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:40:06.481775 (+ 568us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:40:06.482189 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:40:06.482194 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:40:06.482198 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:40:06.482402 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:40:06.483430 (+ 1028us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:40:06.483438 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:40:06.483893 (+ 455us) client_negotiation.cc:769] Sending connection context
0620 18:40:06.484234 (+ 341us) client_negotiation.cc:240] Negotiation successful
0620 18:40:06.484935 (+ 701us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"mutex_wait_us":489,"spinlock_wait_cycles":43136,"thread_start_us":373,"threads_started":1}
W20260620 18:40:06.488651 6870 leader_election.cc:341] T 388ffa0861f04da488834911de7bed8c P 129d88bd84a241a68291203014b805d0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c995f8a8741a4540b650832e93ff4d36 (127.22.37.2:36829): Not found: Tablet not found: 388ffa0861f04da488834911de7bed8c
W20260620 18:40:06.488914 6869 leader_election.cc:341] T 388ffa0861f04da488834911de7bed8c P 129d88bd84a241a68291203014b805d0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 51e1565c5d3a45da9d90ee7d50e5184e (127.22.37.3:35831): Not found: Tablet not found: 388ffa0861f04da488834911de7bed8c
I20260620 18:40:06.489069 6869 leader_election.cc:302] T 388ffa0861f04da488834911de7bed8c P 129d88bd84a241a68291203014b805d0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 129d88bd84a241a68291203014b805d0; no voters: 51e1565c5d3a45da9d90ee7d50e5184e, c995f8a8741a4540b650832e93ff4d36
I20260620 18:40:06.489475 7252 raft_consensus.cc:2726] T 388ffa0861f04da488834911de7bed8c P 129d88bd84a241a68291203014b805d0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 18:40:06.528486 7257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:06.527604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:39147 (local address 127.22.37.3:44703)
0620 18:40:06.528073 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:06.528237 (+ 164us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:39147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":225,"threads_started":1}
I20260620 18:40:06.630191 7254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:06.629705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:39147 (local address 127.22.37.1:36731)
0620 18:40:06.629904 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:06.630017 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:39147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260620 18:40:07.071595 7258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:07.070709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:39147 (local address 127.22.37.2:39269)
0620 18:40:07.071273 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:07.071381 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:39147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":268,"threads_started":1}
I20260620 18:40:07.530210 7259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:07.529461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:39147 (local address 127.22.37.3:33743)
0620 18:40:07.529950 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:07.530053 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:39147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":285,"threads_started":1}
I20260620 18:40:07.631932 7260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:07.631156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:39147 (local address 127.22.37.1:39349)
0620 18:40:07.631649 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:07.631734 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:39147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":235,"threads_started":1}
I20260620 18:40:08.073246 7261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:08.072481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:39147 (local address 127.22.37.2:45761)
0620 18:40:08.072974 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:08.073059 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:39147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":278,"threads_started":1}
I20260620 18:40:08.460242 7262 raft_consensus.cc:482] T 388ffa0861f04da488834911de7bed8c P 129d88bd84a241a68291203014b805d0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:40:08.460459 7262 raft_consensus.cc:504] T 388ffa0861f04da488834911de7bed8c P 129d88bd84a241a68291203014b805d0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c995f8a8741a4540b650832e93ff4d36" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 36829 } } peers { permanent_uuid: "51e1565c5d3a45da9d90ee7d50e5184e" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35831 } } peers { permanent_uuid: "129d88bd84a241a68291203014b805d0" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 42741 } }
I20260620 18:40:08.461145 7262 leader_election.cc:288] T 388ffa0861f04da488834911de7bed8c P 129d88bd84a241a68291203014b805d0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c995f8a8741a4540b650832e93ff4d36 (127.22.37.2:36829), 51e1565c5d3a45da9d90ee7d50e5184e (127.22.37.3:35831)
W20260620 18:40:08.462138 6869 leader_election.cc:341] T 388ffa0861f04da488834911de7bed8c P 129d88bd84a241a68291203014b805d0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 51e1565c5d3a45da9d90ee7d50e5184e (127.22.37.3:35831): Not found: Tablet not found: 388ffa0861f04da488834911de7bed8c
W20260620 18:40:08.462316 6870 leader_election.cc:341] T 388ffa0861f04da488834911de7bed8c P 129d88bd84a241a68291203014b805d0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c995f8a8741a4540b650832e93ff4d36 (127.22.37.2:36829): Not found: Tablet not found: 388ffa0861f04da488834911de7bed8c
I20260620 18:40:08.462431 6870 leader_election.cc:302] T 388ffa0861f04da488834911de7bed8c P 129d88bd84a241a68291203014b805d0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 129d88bd84a241a68291203014b805d0; no voters: 51e1565c5d3a45da9d90ee7d50e5184e, c995f8a8741a4540b650832e93ff4d36
I20260620 18:40:08.462661 7262 raft_consensus.cc:2726] T 388ffa0861f04da488834911de7bed8c P 129d88bd84a241a68291203014b805d0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 18:40:08.531796 7263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:08.531074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:39147 (local address 127.22.37.3:41403)
0620 18:40:08.531553 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:08.531636 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:39147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":252,"threads_started":1}
I20260620 18:40:08.633881 7264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:08.632906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:39147 (local address 127.22.37.1:33673)
0620 18:40:08.633528 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:08.633650 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:39147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":284,"threads_started":1}
I20260620 18:40:09.074875 7265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:09.074168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:39147 (local address 127.22.37.2:46713)
0620 18:40:09.074607 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:09.074691 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:39147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":206,"threads_started":1}
I20260620 18:40:09.533473 7266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:09.532716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:39147 (local address 127.22.37.3:41829)
0620 18:40:09.533138 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:09.533219 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:39147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":200,"threads_started":1}
I20260620 18:40:09.635805 7267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:09.635084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:39147 (local address 127.22.37.1:59875)
0620 18:40:09.635522 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:09.635625 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:39147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":239,"threads_started":1}
I20260620 18:40:10.076566 7268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:10.075849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:39147 (local address 127.22.37.2:59159)
0620 18:40:10.076314 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:10.076398 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:39147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":250,"threads_started":1}
I20260620 18:40:10.534976 7269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:10.534341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:39147 (local address 127.22.37.3:45221)
0620 18:40:10.534748 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:10.534823 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:39147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":216,"threads_started":1}
I20260620 18:40:10.637569 7270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:10.636844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:39147 (local address 127.22.37.1:33651)
0620 18:40:10.637306 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:10.637410 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:39147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":256,"threads_started":1}
I20260620 18:40:10.677812 7271 raft_consensus.cc:482] T 388ffa0861f04da488834911de7bed8c P 129d88bd84a241a68291203014b805d0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:40:10.678093 7271 raft_consensus.cc:504] T 388ffa0861f04da488834911de7bed8c P 129d88bd84a241a68291203014b805d0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c995f8a8741a4540b650832e93ff4d36" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 36829 } } peers { permanent_uuid: "51e1565c5d3a45da9d90ee7d50e5184e" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35831 } } peers { permanent_uuid: "129d88bd84a241a68291203014b805d0" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 42741 } }
I20260620 18:40:10.678828 7271 leader_election.cc:288] T 388ffa0861f04da488834911de7bed8c P 129d88bd84a241a68291203014b805d0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c995f8a8741a4540b650832e93ff4d36 (127.22.37.2:36829), 51e1565c5d3a45da9d90ee7d50e5184e (127.22.37.3:35831)
W20260620 18:40:10.679881 6870 leader_election.cc:341] T 388ffa0861f04da488834911de7bed8c P 129d88bd84a241a68291203014b805d0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c995f8a8741a4540b650832e93ff4d36 (127.22.37.2:36829): Not found: Tablet not found: 388ffa0861f04da488834911de7bed8c
W20260620 18:40:10.680100 6869 leader_election.cc:341] T 388ffa0861f04da488834911de7bed8c P 129d88bd84a241a68291203014b805d0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 51e1565c5d3a45da9d90ee7d50e5184e (127.22.37.3:35831): Not found: Tablet not found: 388ffa0861f04da488834911de7bed8c
I20260620 18:40:10.680245 6869 leader_election.cc:302] T 388ffa0861f04da488834911de7bed8c P 129d88bd84a241a68291203014b805d0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 129d88bd84a241a68291203014b805d0; no voters: 51e1565c5d3a45da9d90ee7d50e5184e, c995f8a8741a4540b650832e93ff4d36
I20260620 18:40:10.680501 7271 raft_consensus.cc:2726] T 388ffa0861f04da488834911de7bed8c P 129d88bd84a241a68291203014b805d0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 18:40:11.078298 7272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:11.077546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:39147 (local address 127.22.37.2:60169)
0620 18:40:11.078008 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:11.078116 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:39147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":245,"threads_started":1}
I20260620 18:40:11.536728 7273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:11.535893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:39147 (local address 127.22.37.3:42669)
0620 18:40:11.536437 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:11.536536 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:39147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":290,"threads_started":1}
I20260620 18:40:11.639194 7274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:11.638470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:39147 (local address 127.22.37.1:37593)
0620 18:40:11.638898 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:11.639006 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:39147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":230,"threads_started":1}
I20260620 18:40:12.080303 7275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:12.079424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:39147 (local address 127.22.37.2:38375)
0620 18:40:12.080010 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:12.080113 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:39147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":293,"threads_started":1}
I20260620 18:40:12.538578 7276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:12.537760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:39147 (local address 127.22.37.3:41819)
0620 18:40:12.538290 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:12.538381 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:39147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":218,"threads_started":1}
I20260620 18:40:12.640972 7277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:12.640217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:39147 (local address 127.22.37.1:49585)
0620 18:40:12.640708 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:12.640809 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:39147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":258,"threads_started":1}
I20260620 18:40:13.082120 7278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:13.081359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:39147 (local address 127.22.37.2:43145)
0620 18:40:13.081844 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:13.081943 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:39147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":209,"threads_started":1}
I20260620 18:40:13.540504 7279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:13.539762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:39147 (local address 127.22.37.3:55491)
0620 18:40:13.540230 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:13.540317 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:39147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":241,"threads_started":1}
I20260620 18:40:13.642825 7280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:13.642002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:39147 (local address 127.22.37.1:60337)
0620 18:40:13.642518 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:13.642643 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:39147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":247,"threads_started":1}
I20260620 18:40:13.733243 7281 raft_consensus.cc:482] T 388ffa0861f04da488834911de7bed8c P 129d88bd84a241a68291203014b805d0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:40:13.733526 7281 raft_consensus.cc:504] T 388ffa0861f04da488834911de7bed8c P 129d88bd84a241a68291203014b805d0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c995f8a8741a4540b650832e93ff4d36" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 36829 } } peers { permanent_uuid: "51e1565c5d3a45da9d90ee7d50e5184e" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35831 } } peers { permanent_uuid: "129d88bd84a241a68291203014b805d0" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 42741 } }
I20260620 18:40:13.734153 7281 leader_election.cc:288] T 388ffa0861f04da488834911de7bed8c P 129d88bd84a241a68291203014b805d0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c995f8a8741a4540b650832e93ff4d36 (127.22.37.2:36829), 51e1565c5d3a45da9d90ee7d50e5184e (127.22.37.3:35831)
W20260620 18:40:13.735069 6869 leader_election.cc:341] T 388ffa0861f04da488834911de7bed8c P 129d88bd84a241a68291203014b805d0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 51e1565c5d3a45da9d90ee7d50e5184e (127.22.37.3:35831): Not found: Tablet not found: 388ffa0861f04da488834911de7bed8c
W20260620 18:40:13.735260 6870 leader_election.cc:341] T 388ffa0861f04da488834911de7bed8c P 129d88bd84a241a68291203014b805d0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c995f8a8741a4540b650832e93ff4d36 (127.22.37.2:36829): Not found: Tablet not found: 388ffa0861f04da488834911de7bed8c
I20260620 18:40:13.735352 6870 leader_election.cc:302] T 388ffa0861f04da488834911de7bed8c P 129d88bd84a241a68291203014b805d0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 129d88bd84a241a68291203014b805d0; no voters: 51e1565c5d3a45da9d90ee7d50e5184e, c995f8a8741a4540b650832e93ff4d36
I20260620 18:40:13.735550 7281 raft_consensus.cc:2726] T 388ffa0861f04da488834911de7bed8c P 129d88bd84a241a68291203014b805d0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 18:40:14.083827 7282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:14.083085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:39147 (local address 127.22.37.2:41245)
0620 18:40:14.083564 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:14.083666 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:39147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":231,"threads_started":1}
I20260620 18:40:14.542325 7283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:14.541529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:39147 (local address 127.22.37.3:44565)
0620 18:40:14.542064 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:14.542167 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:39147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":241,"threads_started":1}
I20260620 18:40:14.644698 7284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:14.643889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:39147 (local address 127.22.37.1:49243)
0620 18:40:14.644444 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:14.644530 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:39147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":331,"threads_started":1}
I20260620 18:40:15.085465 7285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:15.084734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:39147 (local address 127.22.37.2:57163)
0620 18:40:15.085147 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:15.085229 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:39147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":213,"threads_started":1}
I20260620 18:40:15.543917 7286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:15.543277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:39147 (local address 127.22.37.3:40453)
0620 18:40:15.543675 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:15.543752 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:39147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":213,"threads_started":1}
I20260620 18:40:15.646449 7287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:15.645705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:39147 (local address 127.22.37.1:39127)
0620 18:40:15.646185 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:15.646271 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:39147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":248,"threads_started":1}
I20260620 18:40:16.087214 7288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:16.086461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:39147 (local address 127.22.37.2:48843)
0620 18:40:16.086948 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:16.087032 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:39147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":246,"threads_started":1}
I20260620 18:40:16.545639 7289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:16.544798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:39147 (local address 127.22.37.3:51691)
0620 18:40:16.545318 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:16.545462 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:39147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":274,"threads_started":1}
I20260620 18:40:16.648051 7290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:16.647336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:39147 (local address 127.22.37.1:53163)
0620 18:40:16.647794 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:16.647880 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:39147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":224,"threads_started":1}
I20260620 18:40:17.083611 7291 raft_consensus.cc:482] T 388ffa0861f04da488834911de7bed8c P 129d88bd84a241a68291203014b805d0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:40:17.083844 7291 raft_consensus.cc:504] T 388ffa0861f04da488834911de7bed8c P 129d88bd84a241a68291203014b805d0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c995f8a8741a4540b650832e93ff4d36" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 36829 } } peers { permanent_uuid: "51e1565c5d3a45da9d90ee7d50e5184e" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 35831 } } peers { permanent_uuid: "129d88bd84a241a68291203014b805d0" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 42741 } }
I20260620 18:40:17.084553 7291 leader_election.cc:288] T 388ffa0861f04da488834911de7bed8c P 129d88bd84a241a68291203014b805d0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c995f8a8741a4540b650832e93ff4d36 (127.22.37.2:36829), 51e1565c5d3a45da9d90ee7d50e5184e (127.22.37.3:35831)
W20260620 18:40:17.085690 6869 leader_election.cc:341] T 388ffa0861f04da488834911de7bed8c P 129d88bd84a241a68291203014b805d0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 51e1565c5d3a45da9d90ee7d50e5184e (127.22.37.3:35831): Not found: Tablet not found: 388ffa0861f04da488834911de7bed8c
W20260620 18:40:17.085920 6870 leader_election.cc:341] T 388ffa0861f04da488834911de7bed8c P 129d88bd84a241a68291203014b805d0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c995f8a8741a4540b650832e93ff4d36 (127.22.37.2:36829): Not found: Tablet not found: 388ffa0861f04da488834911de7bed8c
I20260620 18:40:17.086071 6870 leader_election.cc:302] T 388ffa0861f04da488834911de7bed8c P 129d88bd84a241a68291203014b805d0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 129d88bd84a241a68291203014b805d0; no voters: 51e1565c5d3a45da9d90ee7d50e5184e, c995f8a8741a4540b650832e93ff4d36
I20260620 18:40:17.086340 7291 raft_consensus.cc:2726] T 388ffa0861f04da488834911de7bed8c P 129d88bd84a241a68291203014b805d0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 18:40:17.088755 7292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:17.088015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:39147 (local address 127.22.37.2:34537)
0620 18:40:17.088510 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:17.088589 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:39147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":285,"threads_started":1}
I20260620 18:40:17.547391 7293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:17.546619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:39147 (local address 127.22.37.3:43179)
0620 18:40:17.547120 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:17.547218 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:39147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":269,"threads_started":1}
I20260620 18:40:17.649837 7294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:17.649032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:39147 (local address 127.22.37.1:41177)
0620 18:40:17.649577 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:17.649659 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:39147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":268,"threads_started":1}
I20260620 18:40:18.090544 7295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:18.089705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:39147 (local address 127.22.37.2:53053)
0620 18:40:18.090292 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:18.090382 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:39147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":392,"threads_started":1}
I20260620 18:40:18.549069 7296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:18.548333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:39147 (local address 127.22.37.3:57033)
0620 18:40:18.548796 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:18.548906 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:39147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":247,"threads_started":1}
I20260620 18:40:18.651512 7297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:18.650739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:39147 (local address 127.22.37.1:52027)
0620 18:40:18.651228 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:18.651311 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:39147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":231,"threads_started":1}
I20260620 18:40:19.092235 7298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:19.091488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:39147 (local address 127.22.37.2:42339)
0620 18:40:19.091943 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:19.092026 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:39147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":250,"threads_started":1}
I20260620 18:40:19.550873 7299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:19.550127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:39147 (local address 127.22.37.3:51845)
0620 18:40:19.550593 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:19.550681 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:39147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":227,"threads_started":1}
I20260620 18:40:19.653903 7300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:19.652921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:39147 (local address 127.22.37.1:52351)
0620 18:40:19.653610 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:19.653723 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:39147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":339,"threads_started":1}
I20260620 18:40:20.094364 7301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:20.093618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:39147 (local address 127.22.37.2:58093)
0620 18:40:20.094104 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:20.094189 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:39147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":245,"threads_started":1}
I20260620 18:40:20.536407 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 6813
I20260620 18:40:20.551479 7302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:20.537659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41696 (local address 127.22.37.1:42741)
0620 18:40:20.538305 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:20.538315 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:20.538349 (+ 34us) server_negotiation.cc:407] Connection header received
0620 18:40:20.538412 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:20.538418 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:20.538642 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:20.538816 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:40:20.540212 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:20.541208 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:20.542510 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:20.543060 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:20.543211 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:20.545925 (+ 2714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:40:20.545935 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:40:20.545950 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:40:20.545983 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:40:20.548645 (+ 2662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:40:20.549306 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:40:20.549313 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:40:20.549316 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:40:20.549500 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:40:20.549862 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:40:20.549867 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:40:20.549870 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:40:20.550236 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:40:20.550569 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:20.550890 (+ 321us) server_negotiation.cc:299] Negotiation successful
0620 18:40:20.551169 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":283,"threads_started":1}
I20260620 18:40:20.552762 7303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:20.552087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:39147 (local address 127.22.37.3:53197)
0620 18:40:20.552524 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:20.552617 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.62:39147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":224,"threads_started":1}
I20260620 18:40:20.552847 6978 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:40:20.641450 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 6813
I20260620 18:40:20.661932 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 6986
I20260620 18:40:20.675159 7305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:20.662703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44488 (local address 127.22.37.2:36829)
0620 18:40:20.663177 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:20.663181 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:20.663197 (+ 16us) server_negotiation.cc:407] Connection header received
0620 18:40:20.663251 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:20.663254 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:20.663313 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:20.663401 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:40:20.664827 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:20.665673 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:20.666915 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:20.667349 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:20.667445 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:20.670106 (+ 2661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:40:20.670112 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:40:20.670115 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:40:20.670150 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:40:20.672705 (+ 2555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:40:20.673345 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:40:20.673349 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:40:20.673351 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:40:20.673555 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:40:20.673889 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:40:20.673892 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:40:20.673894 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:40:20.674162 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:40:20.674354 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:20.674598 (+ 244us) server_negotiation.cc:299] Negotiation successful
0620 18:40:20.674761 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":273,"threads_started":1}
I20260620 18:40:20.675980 7110 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:40:20.758997 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 6986
I20260620 18:40:20.777272 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 7118
I20260620 18:40:20.790840 7307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:20.778321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50438 (local address 127.22.37.3:35831)
0620 18:40:20.778848 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:20.778853 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:20.778868 (+ 15us) server_negotiation.cc:407] Connection header received
0620 18:40:20.778924 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:20.778927 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:20.778991 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:20.779086 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:40:20.780419 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:20.781194 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:20.782566 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:20.783009 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:20.783102 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:20.785649 (+ 2547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:40:20.785654 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:40:20.785658 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:40:20.785690 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:40:20.788429 (+ 2739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:40:20.789027 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:40:20.789030 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:40:20.789032 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:40:20.789188 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:40:20.789628 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:40:20.789632 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:40:20.789634 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:40:20.789891 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:40:20.790094 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:20.790299 (+ 205us) server_negotiation.cc:299] Negotiation successful
0620 18:40:20.790459 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":244,"threads_started":1}
I20260620 18:40:20.791738 7241 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:40:20.875002 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 7118
2026-06-20T18:40:20Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64647 ms)
[----------] 30 tests from SecurityITest (293409 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980527431848-22676-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:40:20 dist-test-slave-5lgp krb5kdc[7311](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:40:20 dist-test-slave-5lgp krb5kdc[7311](info): set up 2 sockets
Jun 20 18:40:20 dist-test-slave-5lgp krb5kdc[7311](info): commencing operation
krb5kdc: starting...
W20260620 18:40:21.993551 22676 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980527431848-22676-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980527431848-22676-0/krb5kdc/test-user.keytab.
Jun 20 18:40:22 dist-test-slave-5lgp krb5kdc[7311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:40:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:40:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.37.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.37.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.37.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab.
Entry for principal kudu/127.22.37.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab.
I20260620 18:40:22.240495 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980527431848-22676-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980527431848-22676-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980527431848-22676-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.62:41813
--webserver_interface=127.22.37.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab
--principal=kudu/127.22.37.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:33661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.37.62:41813
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:40:22.569018 7324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:40:22.569437 7324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:40:22.569628 7324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:40:22.577064 7324 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:40:22.577216 7324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:40:22.577329 7324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:40:22.577450 7324 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:40:22.577539 7324 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:40:22.587651 7324 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:33661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980527431848-22676-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980527431848-22676-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.37.62:41813
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab
--principal=kudu/127.22.37.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.62:41813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.37.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980527431848-22676-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:40:22.589061 7324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:40:22.590413 7324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:40:22.602123 7329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:40:22.602118 7330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:40:22.602311 7332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:40:22.602802 7324 server_base.cc:1034] running on GCE node
I20260620 18:40:22.603514 7324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:40:22.604794 7324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:40:22.606026 7324 hybrid_clock.cc:648] HybridClock initialized: now 1781980822605984 us; error 49 us; skew 500 ppm
I20260620 18:40:22.606366 7324 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:40:22 dist-test-slave-5lgp krb5kdc[7311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980822, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:40:22.609416 7324 init.cc:377] Logged in from keytab as kudu/127.22.37.62@KRBTEST.COM (short username kudu)
I20260620 18:40:22.610808 7324 webserver.cc:460] Webserver started at http://127.22.37.62:35217/ using document root <none> and password file <none>
I20260620 18:40:22.611445 7324 fs_manager.cc:359] Metadata directory not provided
I20260620 18:40:22.611558 7324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:40:22.611800 7324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:40:22.614741 7324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980527431848-22676-0/minicluster-data/master-0/data/instance:
uuid: "fb920b9ac1434a038cf12bccb256f3b3"
format_stamp: "Formatted at 2026-06-20 18:40:22 on dist-test-slave-5lgp"
server_key: "e4b7ebcf5e2814518d2be6b92fc19774"
server_key_iv: "2622c6f841be609abcd496a98692de8d"
server_key_version: "encryptionkey@0"
I20260620 18:40:22.615449 7324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980527431848-22676-0/minicluster-data/master-0/wal/instance:
uuid: "fb920b9ac1434a038cf12bccb256f3b3"
format_stamp: "Formatted at 2026-06-20 18:40:22 on dist-test-slave-5lgp"
server_key: "e4b7ebcf5e2814518d2be6b92fc19774"
server_key_iv: "2622c6f841be609abcd496a98692de8d"
server_key_version: "encryptionkey@0"
I20260620 18:40:22.620422 7324 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 18:40:22.623773 7339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:40:22.625047 7324 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:40:22.625185 7324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980527431848-22676-0/minicluster-data/master-0/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980527431848-22676-0/minicluster-data/master-0/wal
uuid: "fb920b9ac1434a038cf12bccb256f3b3"
format_stamp: "Formatted at 2026-06-20 18:40:22 on dist-test-slave-5lgp"
server_key: "e4b7ebcf5e2814518d2be6b92fc19774"
server_key_iv: "2622c6f841be609abcd496a98692de8d"
server_key_version: "encryptionkey@0"
I20260620 18:40:22.625341 7324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980527431848-22676-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980527431848-22676-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980527431848-22676-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:40:22.643783 7324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:40:22.648099 7324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:40:22.648341 7324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:40:22.666309 7324 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.62:41813
I20260620 18:40:22.666404 7390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.62:41813 every 8 connection(s)
I20260620 18:40:22.667699 7324 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
I20260620 18:40:22.671334 7391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:40:22.673589 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 7324
I20260620 18:40:22.673827 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980527431848-22676-0/minicluster-data/master-0/wal/instance
I20260620 18:40:22.674609 22676 external_mini_cluster.cc:1428] Setting key ce9dc1e574023e7ba701cc9305ebbd5e
I20260620 18:40:22.684321 7391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb920b9ac1434a038cf12bccb256f3b3: Bootstrap starting.
I20260620 18:40:22.688120 7391 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb920b9ac1434a038cf12bccb256f3b3: Neither blocks nor log segments found. Creating new log.
I20260620 18:40:22.689466 7391 log.cc:824] T 00000000000000000000000000000000 P fb920b9ac1434a038cf12bccb256f3b3: Log is configured to *not* fsync() on all Append() calls
Jun 20 18:40:22 dist-test-slave-5lgp krb5kdc[7311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM
I20260620 18:40:22.692308 7391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb920b9ac1434a038cf12bccb256f3b3: No bootstrap required, opened a new log
I20260620 18:40:22.698254 7394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:22.676704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46432 (local address 127.22.37.62:41813)
0620 18:40:22.677583 (+ 879us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:22.677597 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:22.677639 (+ 42us) server_negotiation.cc:407] Connection header received
0620 18:40:22.679346 (+ 1707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:22.679369 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:22.679778 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:22.680079 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:40:22.681823 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:22.683253 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:22.685135 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:22.685855 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:22.685990 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:22.690930 (+ 4940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:40:22.690938 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:40:22.690953 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:40:22.690987 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:40:22.694147 (+ 3160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:40:22.694824 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:40:22.694830 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:40:22.694836 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:40:22.695001 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:40:22.695391 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:40:22.695397 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:40:22.695401 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:40:22.695947 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:40:22.696283 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:22.696789 (+ 506us) server_negotiation.cc:299] Negotiation successful
0620 18:40:22.697208 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":686,"thread_start_us":407,"threads_started":1}
I20260620 18:40:22.700794 7391 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb920b9ac1434a038cf12bccb256f3b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb920b9ac1434a038cf12bccb256f3b3" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 41813 } }
I20260620 18:40:22.701243 7391 raft_consensus.cc:374] T 00000000000000000000000000000000 P fb920b9ac1434a038cf12bccb256f3b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:40:22.701359 7391 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb920b9ac1434a038cf12bccb256f3b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb920b9ac1434a038cf12bccb256f3b3, State: Initialized, Role: FOLLOWER
I20260620 18:40:22.701985 7391 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb920b9ac1434a038cf12bccb256f3b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb920b9ac1434a038cf12bccb256f3b3" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 41813 } }
I20260620 18:40:22.702224 7391 raft_consensus.cc:388] T 00000000000000000000000000000000 P fb920b9ac1434a038cf12bccb256f3b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:40:22.702366 7391 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb920b9ac1434a038cf12bccb256f3b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:40:22.702541 7391 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb920b9ac1434a038cf12bccb256f3b3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:40:22.704604 7391 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb920b9ac1434a038cf12bccb256f3b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb920b9ac1434a038cf12bccb256f3b3" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 41813 } }
I20260620 18:40:22.705081 7391 leader_election.cc:302] T 00000000000000000000000000000000 P fb920b9ac1434a038cf12bccb256f3b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb920b9ac1434a038cf12bccb256f3b3; no voters:
I20260620 18:40:22.705705 7391 leader_election.cc:288] T 00000000000000000000000000000000 P fb920b9ac1434a038cf12bccb256f3b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:40:22.705827 7396 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb920b9ac1434a038cf12bccb256f3b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:40:22.706545 7396 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb920b9ac1434a038cf12bccb256f3b3 [term 1 LEADER]: Becoming Leader. State: Replica: fb920b9ac1434a038cf12bccb256f3b3, State: Running, Role: LEADER
I20260620 18:40:22.707085 7396 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb920b9ac1434a038cf12bccb256f3b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb920b9ac1434a038cf12bccb256f3b3" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 41813 } }
I20260620 18:40:22.707929 7391 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb920b9ac1434a038cf12bccb256f3b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:40:22.710125 7397 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb920b9ac1434a038cf12bccb256f3b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fb920b9ac1434a038cf12bccb256f3b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb920b9ac1434a038cf12bccb256f3b3" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 41813 } } }
I20260620 18:40:22.710083 7398 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb920b9ac1434a038cf12bccb256f3b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb920b9ac1434a038cf12bccb256f3b3. Latest consensus state: current_term: 1 leader_uuid: "fb920b9ac1434a038cf12bccb256f3b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb920b9ac1434a038cf12bccb256f3b3" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 41813 } } }
I20260620 18:40:22.710481 7397 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb920b9ac1434a038cf12bccb256f3b3 [sys.catalog]: This master's current role is: LEADER
I20260620 18:40:22.710511 7398 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb920b9ac1434a038cf12bccb256f3b3 [sys.catalog]: This master's current role is: LEADER
I20260620 18:40:22.717962 7403 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:40:22.723263 7403 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:40:22.731971 7403 catalog_manager.cc:1347] Generated new cluster ID: 263d40d7b3fa4a9b93fb2b9b512fdf83
I20260620 18:40:22.732088 7403 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:40:22.755185 7403 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:40:22.756517 7403 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:40:22.777275 7403 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fb920b9ac1434a038cf12bccb256f3b3: Generated new TSK 0
I20260620 18:40:22.778170 7403 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.37.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.37.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.37.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.1.keytab.
Entry for principal kudu/127.22.37.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.1.keytab.
I20260620 18:40:22.845280 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980527431848-22676-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980527431848-22676-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980527431848-22676-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980527431848-22676-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.37.1:0
--local_ip_for_outbound_sockets=127.22.37.1
--webserver_interface=127.22.37.1
--webserver_port=0
--tserver_master_addrs=127.22.37.62:41813
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.1.keytab
--principal=kudu/127.22.37.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:33661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:40:23.176225 7417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:40:23.176548 7417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:40:23.176721 7417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:40:23.183894 7417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:40:23.184028 7417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:40:23.184207 7417 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.37.1
I20260620 18:40:23.193765 7417 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:33661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980527431848-22676-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980527431848-22676-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.1.keytab
--principal=kudu/127.22.37.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.37.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980527431848-22676-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.37.1
--webserver_port=0
--tserver_master_addrs=127.22.37.62:41813
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.37.1
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980527431848-22676-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:40:23.195030 7417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:40:23.196336 7417 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:40:23.209786 7422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:40:23.209687 7425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:40:23.209713 7423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:40:23.210521 7417 server_base.cc:1034] running on GCE node
I20260620 18:40:23.211102 7417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:40:23.212051 7417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:40:23.213289 7417 hybrid_clock.cc:648] HybridClock initialized: now 1781980823213258 us; error 49 us; skew 500 ppm
I20260620 18:40:23.213639 7417 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:40:23 dist-test-slave-5lgp krb5kdc[7311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980823, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:40:23.216809 7417 init.cc:377] Logged in from keytab as kudu/127.22.37.1@KRBTEST.COM (short username kudu)
I20260620 18:40:23.218303 7417 webserver.cc:460] Webserver started at http://127.22.37.1:39921/ using document root <none> and password file <none>
I20260620 18:40:23.219022 7417 fs_manager.cc:359] Metadata directory not provided
I20260620 18:40:23.219172 7417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:40:23.219504 7417 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:40:23.222391 7417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980527431848-22676-0/minicluster-data/ts-0/data/instance:
uuid: "23a2859f241b4df582667728cb114d0d"
format_stamp: "Formatted at 2026-06-20 18:40:23 on dist-test-slave-5lgp"
server_key: "a8de5865ca6c4325b5d89b5fb3c9cee6"
server_key_iv: "8bf151767da865eaf6d7a70f08590780"
server_key_version: "encryptionkey@0"
I20260620 18:40:23.223124 7417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980527431848-22676-0/minicluster-data/ts-0/wal/instance:
uuid: "23a2859f241b4df582667728cb114d0d"
format_stamp: "Formatted at 2026-06-20 18:40:23 on dist-test-slave-5lgp"
server_key: "a8de5865ca6c4325b5d89b5fb3c9cee6"
server_key_iv: "8bf151767da865eaf6d7a70f08590780"
server_key_version: "encryptionkey@0"
I20260620 18:40:23.228052 7417 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 18:40:23.231367 7432 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:40:23.232620 7417 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:40:23.232793 7417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980527431848-22676-0/minicluster-data/ts-0/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980527431848-22676-0/minicluster-data/ts-0/wal
uuid: "23a2859f241b4df582667728cb114d0d"
format_stamp: "Formatted at 2026-06-20 18:40:23 on dist-test-slave-5lgp"
server_key: "a8de5865ca6c4325b5d89b5fb3c9cee6"
server_key_iv: "8bf151767da865eaf6d7a70f08590780"
server_key_version: "encryptionkey@0"
I20260620 18:40:23.232985 7417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980527431848-22676-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980527431848-22676-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980527431848-22676-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:40:23.253225 7417 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:40:23.257766 7417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:40:23.258077 7417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:40:23.259091 7417 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:40:23.260445 7417 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:40:23.260550 7417 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:40:23.260713 7417 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:40:23.260788 7417 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:40:23.293821 7417 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.1:35813
I20260620 18:40:23.293874 7544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.1:35813 every 8 connection(s)
I20260620 18:40:23.295142 7417 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980527431848-22676-0/minicluster-data/ts-0/data/info.pb
I20260620 18:40:23.301555 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 7417
I20260620 18:40:23.301788 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980527431848-22676-0/minicluster-data/ts-0/wal/instance
I20260620 18:40:23.302453 22676 external_mini_cluster.cc:1428] Setting key 82f4724fe046690f9ff2b17599e3e4cc
Jun 20 18:40:23 dist-test-slave-5lgp krb5kdc[7311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980823, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.1@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM
I20260620 18:40:23.317624 7547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:23.297713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.1:41303 (local address 127.22.37.62:41813)
0620 18:40:23.298156 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:23.298164 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:23.299480 (+ 1316us) server_negotiation.cc:407] Connection header received
0620 18:40:23.301401 (+ 1921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:23.301408 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:23.301513 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:23.301652 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:40:23.303400 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:23.304341 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:23.305967 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:23.306725 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:23.306870 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:23.310614 (+ 3744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:40:23.310622 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:40:23.310630 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:40:23.310675 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:40:23.313975 (+ 3300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:40:23.314979 (+ 1004us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:40:23.314983 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:40:23.314986 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:40:23.315123 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:40:23.315669 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:40:23.315674 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:40:23.315677 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:40:23.315982 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:40:23.316099 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:23.317024 (+ 925us) server_negotiation.cc:299] Negotiation successful
0620 18:40:23.317206 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":215,"threads_started":1}
I20260620 18:40:23.319090 7548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:23.298538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:41813 (local address 127.22.37.1:41303)
0620 18:40:23.299295 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:23.299335 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 18:40:23.301081 (+ 1746us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:40:23.301941 (+ 860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:40:23.301950 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:40:23.302510 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:40:23.303206 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:23.303234 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:23.304526 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:23.304530 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:23.305825 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:23.305837 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:23.306952 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:23.306957 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:23.307166 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:23.307852 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:40:23.307879 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:40:23.310387 (+ 2508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:40:23.314122 (+ 3735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:40:23.314133 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:40:23.314154 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:40:23.314828 (+ 674us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:40:23.315235 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:40:23.315243 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:40:23.315250 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:40:23.315498 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:40:23.316119 (+ 621us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:40:23.316130 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:40:23.316612 (+ 482us) client_negotiation.cc:769] Sending connection context
0620 18:40:23.316915 (+ 303us) client_negotiation.cc:240] Negotiation successful
0620 18:40:23.317538 (+ 623us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":285,"threads_started":1}
I20260620 18:40:23.320802 7545 heartbeater.cc:347] Connected to a master server at 127.22.37.62:41813
I20260620 18:40:23.321115 7545 heartbeater.cc:464] Registering TS with master...
I20260620 18:40:23.321908 7545 heartbeater.cc:511] Master 127.22.37.62:41813 requested a full tablet report, sending...
I20260620 18:40:23.323989 7356 ts_manager.cc:194] Registered new tserver with Master: 23a2859f241b4df582667728cb114d0d (127.22.37.1:35813)
I20260620 18:40:23.325592 7356 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.37.1@KRBTEST.COM'} at 127.22.37.1:41303
WARNING: no policy specified for kudu/127.22.37.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.37.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.37.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.2.keytab.
Entry for principal kudu/127.22.37.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.2.keytab.
I20260620 18:40:23.358493 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980527431848-22676-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980527431848-22676-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980527431848-22676-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980527431848-22676-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.37.2:0
--local_ip_for_outbound_sockets=127.22.37.2
--webserver_interface=127.22.37.2
--webserver_port=0
--tserver_master_addrs=127.22.37.62:41813
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.2.keytab
--principal=kudu/127.22.37.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:33661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:40:23.682574 7551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:40:23.682950 7551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:40:23.683089 7551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:40:23.692751 7551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:40:23.692848 7551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:40:23.692957 7551 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.37.2
I20260620 18:40:23.702623 7551 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:33661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980527431848-22676-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980527431848-22676-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.2.keytab
--principal=kudu/127.22.37.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.37.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980527431848-22676-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.37.2
--webserver_port=0
--tserver_master_addrs=127.22.37.62:41813
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.37.2
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980527431848-22676-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:40:23.703773 7551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:40:23.704988 7551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:40:23.718086 7556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:40:23.718112 7559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:40:23.718138 7557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:40:23.719094 7551 server_base.cc:1034] running on GCE node
I20260620 18:40:23.719622 7551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:40:23.720446 7551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:40:23.721675 7551 hybrid_clock.cc:648] HybridClock initialized: now 1781980823721610 us; error 55 us; skew 500 ppm
I20260620 18:40:23.721994 7551 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:40:23 dist-test-slave-5lgp krb5kdc[7311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980823, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:40:23.725147 7551 init.cc:377] Logged in from keytab as kudu/127.22.37.2@KRBTEST.COM (short username kudu)
I20260620 18:40:23.726619 7551 webserver.cc:460] Webserver started at http://127.22.37.2:37487/ using document root <none> and password file <none>
I20260620 18:40:23.727294 7551 fs_manager.cc:359] Metadata directory not provided
I20260620 18:40:23.727432 7551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:40:23.727743 7551 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:40:23.730752 7551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980527431848-22676-0/minicluster-data/ts-1/data/instance:
uuid: "f2cd8925bb0a47c5adb02808b676d77e"
format_stamp: "Formatted at 2026-06-20 18:40:23 on dist-test-slave-5lgp"
server_key: "622ac6184ff88b3b11f15d3d93f02818"
server_key_iv: "dfae64c51933e271450223d2670d4a45"
server_key_version: "encryptionkey@0"
I20260620 18:40:23.731483 7551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980527431848-22676-0/minicluster-data/ts-1/wal/instance:
uuid: "f2cd8925bb0a47c5adb02808b676d77e"
format_stamp: "Formatted at 2026-06-20 18:40:23 on dist-test-slave-5lgp"
server_key: "622ac6184ff88b3b11f15d3d93f02818"
server_key_iv: "dfae64c51933e271450223d2670d4a45"
server_key_version: "encryptionkey@0"
I20260620 18:40:23.736578 7551 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 18:40:23.740234 7566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:40:23.741544 7551 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:40:23.741719 7551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980527431848-22676-0/minicluster-data/ts-1/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980527431848-22676-0/minicluster-data/ts-1/wal
uuid: "f2cd8925bb0a47c5adb02808b676d77e"
format_stamp: "Formatted at 2026-06-20 18:40:23 on dist-test-slave-5lgp"
server_key: "622ac6184ff88b3b11f15d3d93f02818"
server_key_iv: "dfae64c51933e271450223d2670d4a45"
server_key_version: "encryptionkey@0"
I20260620 18:40:23.741930 7551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980527431848-22676-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980527431848-22676-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980527431848-22676-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:40:23.754148 7551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:40:23.758288 7551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:40:23.758561 7551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:40:23.759442 7551 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:40:23.760771 7551 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:40:23.760871 7551 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:40:23.761010 7551 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:40:23.761085 7551 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:40:23.794487 7551 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.2:38573
I20260620 18:40:23.794519 7678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.2:38573 every 8 connection(s)
I20260620 18:40:23.795760 7551 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980527431848-22676-0/minicluster-data/ts-1/data/info.pb
I20260620 18:40:23.800558 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 7551
I20260620 18:40:23.800770 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980527431848-22676-0/minicluster-data/ts-1/wal/instance
I20260620 18:40:23.801208 22676 external_mini_cluster.cc:1428] Setting key 4800ec3265d2a1113bdb7717b9da0232
Jun 20 18:40:23 dist-test-slave-5lgp krb5kdc[7311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980823, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.2@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM
I20260620 18:40:23.815279 7547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:23.798304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.2:43293 (local address 127.22.37.62:41813)
0620 18:40:23.798490 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:23.798494 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:23.799684 (+ 1190us) server_negotiation.cc:407] Connection header received
0620 18:40:23.801655 (+ 1971us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:23.801660 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:23.801750 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:23.801869 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:40:23.803296 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:23.803993 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:23.805622 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:23.806036 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:23.806184 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:23.809637 (+ 3453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:40:23.809641 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:40:23.809645 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:40:23.809675 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:40:23.812224 (+ 2549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:40:23.812933 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:40:23.812938 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:40:23.812940 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:40:23.813065 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:40:23.813568 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:40:23.813573 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:40:23.813576 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:40:23.813872 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:40:23.814034 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:23.814696 (+ 662us) server_negotiation.cc:299] Negotiation successful
0620 18:40:23.814925 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 18:40:23.816227 7681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:23.798843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:41813 (local address 127.22.37.2:43293)
0620 18:40:23.799514 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:23.799554 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 18:40:23.801302 (+ 1748us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:40:23.802100 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:40:23.802109 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:40:23.802596 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:40:23.803126 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:23.803145 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:23.804146 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:23.804151 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:23.805480 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:23.805491 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:23.806251 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:23.806257 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:23.806489 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:23.807158 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:40:23.807179 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:40:23.809427 (+ 2248us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:40:23.812368 (+ 2941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:40:23.812376 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:40:23.812391 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:40:23.812801 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:40:23.813193 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:40:23.813198 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:40:23.813202 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:40:23.813444 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:40:23.813997 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:40:23.814005 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:40:23.814339 (+ 334us) client_negotiation.cc:769] Sending connection context
0620 18:40:23.814589 (+ 250us) client_negotiation.cc:240] Negotiation successful
0620 18:40:23.815099 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":271,"threads_started":1}
I20260620 18:40:23.817699 7679 heartbeater.cc:347] Connected to a master server at 127.22.37.62:41813
I20260620 18:40:23.818027 7679 heartbeater.cc:464] Registering TS with master...
I20260620 18:40:23.818786 7679 heartbeater.cc:511] Master 127.22.37.62:41813 requested a full tablet report, sending...
I20260620 18:40:23.820310 7356 ts_manager.cc:194] Registered new tserver with Master: f2cd8925bb0a47c5adb02808b676d77e (127.22.37.2:38573)
I20260620 18:40:23.821646 7356 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.37.2@KRBTEST.COM'} at 127.22.37.2:43293
WARNING: no policy specified for kudu/127.22.37.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.37.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.37.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.3.keytab.
Entry for principal kudu/127.22.37.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.3.keytab.
I20260620 18:40:23.854466 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980527431848-22676-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980527431848-22676-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980527431848-22676-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980527431848-22676-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.37.3:0
--local_ip_for_outbound_sockets=127.22.37.3
--webserver_interface=127.22.37.3
--webserver_port=0
--tserver_master_addrs=127.22.37.62:41813
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.3.keytab
--principal=kudu/127.22.37.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:33661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:40:24.175146 7684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:40:24.175420 7684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:40:24.175575 7684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:40:24.182835 7684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:40:24.182931 7684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:40:24.183065 7684 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.37.3
I20260620 18:40:24.192492 7684 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:33661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980527431848-22676-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980527431848-22676-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.3.keytab
--principal=kudu/127.22.37.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.37.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980527431848-22676-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.37.3
--webserver_port=0
--tserver_master_addrs=127.22.37.62:41813
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.37.3
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980527431848-22676-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:40:24.193735 7684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:40:24.195040 7684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:40:24.208369 7689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:40:24.208453 7692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:40:24.208737 7690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:40:24.209120 7684 server_base.cc:1034] running on GCE node
I20260620 18:40:24.209872 7684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:40:24.210800 7684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:40:24.212000 7684 hybrid_clock.cc:648] HybridClock initialized: now 1781980824211966 us; error 35 us; skew 500 ppm
I20260620 18:40:24.212322 7684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:40:24 dist-test-slave-5lgp krb5kdc[7311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980824, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:40:24.215534 7684 init.cc:377] Logged in from keytab as kudu/127.22.37.3@KRBTEST.COM (short username kudu)
I20260620 18:40:24.217028 7684 webserver.cc:460] Webserver started at http://127.22.37.3:45021/ using document root <none> and password file <none>
I20260620 18:40:24.217816 7684 fs_manager.cc:359] Metadata directory not provided
I20260620 18:40:24.217949 7684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:40:24.218288 7684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:40:24.221118 7684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980527431848-22676-0/minicluster-data/ts-2/data/instance:
uuid: "a97a6d07bc5e401781e977f7ef16f117"
format_stamp: "Formatted at 2026-06-20 18:40:24 on dist-test-slave-5lgp"
server_key: "cc7f0c16be7fbde4a1c2112c367ca3cc"
server_key_iv: "cd08187f6bfe43a7eff7f7418fc75176"
server_key_version: "encryptionkey@0"
I20260620 18:40:24.221897 7684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980527431848-22676-0/minicluster-data/ts-2/wal/instance:
uuid: "a97a6d07bc5e401781e977f7ef16f117"
format_stamp: "Formatted at 2026-06-20 18:40:24 on dist-test-slave-5lgp"
server_key: "cc7f0c16be7fbde4a1c2112c367ca3cc"
server_key_iv: "cd08187f6bfe43a7eff7f7418fc75176"
server_key_version: "encryptionkey@0"
I20260620 18:40:24.226936 7684 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260620 18:40:24.230470 7699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:40:24.231817 7684 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 18:40:24.231983 7684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980527431848-22676-0/minicluster-data/ts-2/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980527431848-22676-0/minicluster-data/ts-2/wal
uuid: "a97a6d07bc5e401781e977f7ef16f117"
format_stamp: "Formatted at 2026-06-20 18:40:24 on dist-test-slave-5lgp"
server_key: "cc7f0c16be7fbde4a1c2112c367ca3cc"
server_key_iv: "cd08187f6bfe43a7eff7f7418fc75176"
server_key_version: "encryptionkey@0"
I20260620 18:40:24.232190 7684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980527431848-22676-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980527431848-22676-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980527431848-22676-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:40:24.263705 7684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:40:24.267750 7684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:40:24.268031 7684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:40:24.268991 7684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:40:24.270288 7684 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:40:24.270391 7684 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:40:24.270548 7684 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:40:24.270622 7684 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:40:24.303504 7684 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.3:33431
I20260620 18:40:24.303553 7811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.3:33431 every 8 connection(s)
I20260620 18:40:24.304775 7684 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980527431848-22676-0/minicluster-data/ts-2/data/info.pb
I20260620 18:40:24.307790 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 7684
I20260620 18:40:24.307988 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980527431848-22676-0/minicluster-data/ts-2/wal/instance
I20260620 18:40:24.308651 22676 external_mini_cluster.cc:1428] Setting key e655263c945597ce8be83b061c5689e6
Jun 20 18:40:24 dist-test-slave-5lgp krb5kdc[7311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980824, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.3@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM
I20260620 18:40:24.325706 7547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:24.307362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.3:38537 (local address 127.22.37.62:41813)
0620 18:40:24.307584 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:24.307588 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:24.308970 (+ 1382us) server_negotiation.cc:407] Connection header received
0620 18:40:24.311173 (+ 2203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:24.311179 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:24.311276 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:24.311385 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:40:24.313017 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:24.313840 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:24.315498 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:24.315919 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:24.316057 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:24.319733 (+ 3676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:40:24.319739 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:40:24.319742 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:40:24.319776 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:40:24.322507 (+ 2731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:40:24.323317 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:40:24.323325 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:40:24.323331 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:40:24.323479 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:40:24.323900 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:40:24.323906 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:40:24.323909 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:40:24.324276 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:40:24.324433 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:24.325114 (+ 681us) server_negotiation.cc:299] Negotiation successful
0620 18:40:24.325333 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 18:40:24.326730 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:24.307888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:41813 (local address 127.22.37.3:38537)
0620 18:40:24.308812 (+ 924us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:24.308856 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 18:40:24.310906 (+ 2050us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:40:24.311797 (+ 891us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:40:24.311805 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:40:24.312309 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:40:24.312853 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:24.312870 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:24.313995 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:24.314000 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:24.315348 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:24.315359 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:24.316151 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:24.316160 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:24.316404 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:24.317112 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:40:24.317137 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:40:24.319495 (+ 2358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:40:24.322632 (+ 3137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:40:24.322641 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:40:24.322654 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:40:24.323192 (+ 538us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:40:24.323611 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:40:24.323615 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:40:24.323617 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:40:24.323786 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:40:24.324405 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:40:24.324413 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:40:24.324759 (+ 346us) client_negotiation.cc:769] Sending connection context
0620 18:40:24.325030 (+ 271us) client_negotiation.cc:240] Negotiation successful
0620 18:40:24.325590 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":329,"threads_started":1}
I20260620 18:40:24.328085 7812 heartbeater.cc:347] Connected to a master server at 127.22.37.62:41813
I20260620 18:40:24.328398 7812 heartbeater.cc:464] Registering TS with master...
I20260620 18:40:24.328656 7545 heartbeater.cc:503] Master 127.22.37.62:41813 was elected leader, sending a full tablet report...
I20260620 18:40:24.329130 7812 heartbeater.cc:511] Master 127.22.37.62:41813 requested a full tablet report, sending...
I20260620 18:40:24.330535 7356 ts_manager.cc:194] Registered new tserver with Master: a97a6d07bc5e401781e977f7ef16f117 (127.22.37.3:33431)
I20260620 18:40:24.331310 7356 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.37.3@KRBTEST.COM'} at 127.22.37.3:38537
I20260620 18:40:24.336586 22676 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:40:24.353226 7547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:24.341110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46438 (local address 127.22.37.62:41813)
0620 18:40:24.341300 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:24.341305 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:24.341830 (+ 525us) server_negotiation.cc:407] Connection header received
0620 18:40:24.341957 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:24.341961 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:24.342020 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:24.342103 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:40:24.343473 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:24.344182 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:24.345652 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:24.346043 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:24.346137 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:24.348358 (+ 2221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:40:24.348362 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:40:24.348365 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:40:24.348394 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:40:24.351047 (+ 2653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:40:24.351659 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:40:24.351662 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:40:24.351664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:40:24.351778 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:40:24.352088 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:40:24.352091 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:40:24.352093 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:40:24.352382 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:40:24.352566 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:24.352857 (+ 291us) server_negotiation.cc:299] Negotiation successful
0620 18:40:24.353011 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 18:40:24.356971 7356 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46438:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:40:24.363729 7356 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:40:24.386961 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:24.374599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.1:35813 (local address 127.0.0.1:43112)
0620 18:40:24.375623 (+ 1024us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:24.375666 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 18:40:24.375879 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:40:24.379874 (+ 3995us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:40:24.379881 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:40:24.379914 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:40:24.380362 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:24.380379 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:24.382282 (+ 1903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:24.382287 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:24.384121 (+ 1834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:24.384141 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:24.385945 (+ 1804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:24.385952 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:24.386127 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:24.386166 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 18:40:24.386233 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 18:40:24.386368 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":704,"mutex_wait_us":5,"thread_start_us":1628,"threads_started":1}
I20260620 18:40:24.386916 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:24.374748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.3:33431 (local address 127.0.0.1:58738)
0620 18:40:24.376551 (+ 1803us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:24.376572 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 18:40:24.376703 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:40:24.379518 (+ 2815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:40:24.379523 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:40:24.379547 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:40:24.379950 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:24.379967 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:24.381692 (+ 1725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:24.381699 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:24.384504 (+ 2805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:24.384513 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:24.385782 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:24.385787 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:24.385974 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:24.386031 (+ 57us) client_negotiation.cc:769] Sending connection context
0620 18:40:24.386175 (+ 144us) client_negotiation.cc:240] Negotiation successful
0620 18:40:24.386374 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1651,"mutex_wait_us":176,"thread_start_us":1483,"threads_started":1}
I20260620 18:40:24.387305 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:24.374917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43112 (local address 127.22.37.1:35813)
0620 18:40:24.379327 (+ 4410us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:24.379335 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:24.379365 (+ 30us) server_negotiation.cc:407] Connection header received
0620 18:40:24.379455 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:24.379462 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:24.379678 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:24.379865 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:40:24.380759 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:24.382150 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:24.384301 (+ 2151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:24.385710 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:24.386477 (+ 767us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:24.386696 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:24.386774 (+ 78us) server_negotiation.cc:299] Negotiation successful
0620 18:40:24.386947 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4251,"thread_start_us":4073,"threads_started":1}
I20260620 18:40:24.387466 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:24.375925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41562 (local address 127.22.37.2:38573)
0620 18:40:24.377694 (+ 1769us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:24.377707 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:24.378337 (+ 630us) server_negotiation.cc:407] Connection header received
0620 18:40:24.378514 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:24.378524 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:24.378816 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:24.379677 (+ 861us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:40:24.380150 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:24.381755 (+ 1605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:24.384633 (+ 2878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:24.386041 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:24.386174 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:24.386365 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:24.387010 (+ 645us) server_negotiation.cc:299] Negotiation successful
0620 18:40:24.387196 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1542,"thread_start_us":1416,"threads_started":1}
I20260620 18:40:24.387992 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:24.375379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.2:38573 (local address 127.0.0.1:41562)
0620 18:40:24.378198 (+ 2819us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:24.378219 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 18:40:24.378359 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:40:24.379076 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:40:24.379085 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:40:24.379118 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:40:24.379478 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:24.379490 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:24.381886 (+ 2396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:24.381891 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:24.384473 (+ 2582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:24.384492 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:24.386248 (+ 1756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:24.386254 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:24.386426 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:24.386459 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 18:40:24.386643 (+ 184us) client_negotiation.cc:240] Negotiation successful
0620 18:40:24.386773 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2654,"mutex_wait_us":79,"thread_start_us":2498,"threads_started":1}
I20260620 18:40:24.388729 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:24.375396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58738 (local address 127.22.37.3:33431)
0620 18:40:24.379035 (+ 3639us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:24.379043 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:24.379073 (+ 30us) server_negotiation.cc:407] Connection header received
0620 18:40:24.379170 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:24.379178 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:24.379346 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:24.379526 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:40:24.380248 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:24.381555 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:24.384671 (+ 3116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:24.385597 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:24.385740 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:24.385972 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:24.388231 (+ 2259us) server_negotiation.cc:299] Negotiation successful
0620 18:40:24.388409 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3437,"thread_start_us":3258,"threads_started":1}
I20260620 18:40:24.391253 7614 tablet_service.cc:1467] Processing CreateTablet for tablet dd6d00136e144074b235b6bc06ba3bbd (DEFAULT_TABLE table=test-table [id=5c29b8552dfe437e94cf8376762968de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:40:24.391369 7480 tablet_service.cc:1467] Processing CreateTablet for tablet dd6d00136e144074b235b6bc06ba3bbd (DEFAULT_TABLE table=test-table [id=5c29b8552dfe437e94cf8376762968de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:40:24.392439 7614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd6d00136e144074b235b6bc06ba3bbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:40:24.391950 7747 tablet_service.cc:1467] Processing CreateTablet for tablet dd6d00136e144074b235b6bc06ba3bbd (DEFAULT_TABLE table=test-table [id=5c29b8552dfe437e94cf8376762968de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:40:24.392755 7480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd6d00136e144074b235b6bc06ba3bbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:40:24.394109 7747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd6d00136e144074b235b6bc06ba3bbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:40:24.404276 7830 tablet_bootstrap.cc:492] T dd6d00136e144074b235b6bc06ba3bbd P f2cd8925bb0a47c5adb02808b676d77e: Bootstrap starting.
I20260620 18:40:24.408437 7832 tablet_bootstrap.cc:492] T dd6d00136e144074b235b6bc06ba3bbd P a97a6d07bc5e401781e977f7ef16f117: Bootstrap starting.
I20260620 18:40:24.409245 7830 tablet_bootstrap.cc:654] T dd6d00136e144074b235b6bc06ba3bbd P f2cd8925bb0a47c5adb02808b676d77e: Neither blocks nor log segments found. Creating new log.
I20260620 18:40:24.409802 7831 tablet_bootstrap.cc:492] T dd6d00136e144074b235b6bc06ba3bbd P 23a2859f241b4df582667728cb114d0d: Bootstrap starting.
I20260620 18:40:24.410563 7830 log.cc:824] T dd6d00136e144074b235b6bc06ba3bbd P f2cd8925bb0a47c5adb02808b676d77e: Log is configured to *not* fsync() on all Append() calls
I20260620 18:40:24.412161 7832 tablet_bootstrap.cc:654] T dd6d00136e144074b235b6bc06ba3bbd P a97a6d07bc5e401781e977f7ef16f117: Neither blocks nor log segments found. Creating new log.
I20260620 18:40:24.413178 7830 tablet_bootstrap.cc:492] T dd6d00136e144074b235b6bc06ba3bbd P f2cd8925bb0a47c5adb02808b676d77e: No bootstrap required, opened a new log
I20260620 18:40:24.413189 7831 tablet_bootstrap.cc:654] T dd6d00136e144074b235b6bc06ba3bbd P 23a2859f241b4df582667728cb114d0d: Neither blocks nor log segments found. Creating new log.
I20260620 18:40:24.413499 7830 ts_tablet_manager.cc:1397] T dd6d00136e144074b235b6bc06ba3bbd P f2cd8925bb0a47c5adb02808b676d77e: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260620 18:40:24.413461 7832 log.cc:824] T dd6d00136e144074b235b6bc06ba3bbd P a97a6d07bc5e401781e977f7ef16f117: Log is configured to *not* fsync() on all Append() calls
I20260620 18:40:24.414403 7831 log.cc:824] T dd6d00136e144074b235b6bc06ba3bbd P 23a2859f241b4df582667728cb114d0d: Log is configured to *not* fsync() on all Append() calls
I20260620 18:40:24.416438 7832 tablet_bootstrap.cc:492] T dd6d00136e144074b235b6bc06ba3bbd P a97a6d07bc5e401781e977f7ef16f117: No bootstrap required, opened a new log
I20260620 18:40:24.416694 7832 ts_tablet_manager.cc:1397] T dd6d00136e144074b235b6bc06ba3bbd P a97a6d07bc5e401781e977f7ef16f117: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260620 18:40:24.423476 7831 tablet_bootstrap.cc:492] T dd6d00136e144074b235b6bc06ba3bbd P 23a2859f241b4df582667728cb114d0d: No bootstrap required, opened a new log
I20260620 18:40:24.423741 7831 ts_tablet_manager.cc:1397] T dd6d00136e144074b235b6bc06ba3bbd P 23a2859f241b4df582667728cb114d0d: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.004s
I20260620 18:40:24.424081 7830 raft_consensus.cc:348] T dd6d00136e144074b235b6bc06ba3bbd P f2cd8925bb0a47c5adb02808b676d77e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23a2859f241b4df582667728cb114d0d" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 35813 } } peers { permanent_uuid: "a97a6d07bc5e401781e977f7ef16f117" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 33431 } } peers { permanent_uuid: "f2cd8925bb0a47c5adb02808b676d77e" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 38573 } }
I20260620 18:40:24.424522 7830 raft_consensus.cc:374] T dd6d00136e144074b235b6bc06ba3bbd P f2cd8925bb0a47c5adb02808b676d77e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:40:24.424615 7830 raft_consensus.cc:729] T dd6d00136e144074b235b6bc06ba3bbd P f2cd8925bb0a47c5adb02808b676d77e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2cd8925bb0a47c5adb02808b676d77e, State: Initialized, Role: FOLLOWER
I20260620 18:40:24.425236 7830 consensus_queue.cc:260] T dd6d00136e144074b235b6bc06ba3bbd P f2cd8925bb0a47c5adb02808b676d77e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23a2859f241b4df582667728cb114d0d" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 35813 } } peers { permanent_uuid: "a97a6d07bc5e401781e977f7ef16f117" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 33431 } } peers { permanent_uuid: "f2cd8925bb0a47c5adb02808b676d77e" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 38573 } }
I20260620 18:40:24.427578 7679 heartbeater.cc:503] Master 127.22.37.62:41813 was elected leader, sending a full tablet report...
I20260620 18:40:24.427501 7832 raft_consensus.cc:348] T dd6d00136e144074b235b6bc06ba3bbd P a97a6d07bc5e401781e977f7ef16f117 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23a2859f241b4df582667728cb114d0d" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 35813 } } peers { permanent_uuid: "a97a6d07bc5e401781e977f7ef16f117" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 33431 } } peers { permanent_uuid: "f2cd8925bb0a47c5adb02808b676d77e" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 38573 } }
I20260620 18:40:24.427915 7832 raft_consensus.cc:374] T dd6d00136e144074b235b6bc06ba3bbd P a97a6d07bc5e401781e977f7ef16f117 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:40:24.428012 7832 raft_consensus.cc:729] T dd6d00136e144074b235b6bc06ba3bbd P a97a6d07bc5e401781e977f7ef16f117 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97a6d07bc5e401781e977f7ef16f117, State: Initialized, Role: FOLLOWER
I20260620 18:40:24.428288 7830 ts_tablet_manager.cc:1428] T dd6d00136e144074b235b6bc06ba3bbd P f2cd8925bb0a47c5adb02808b676d77e: Time spent starting tablet: real 0.015s user 0.010s sys 0.004s
I20260620 18:40:24.428761 7832 consensus_queue.cc:260] T dd6d00136e144074b235b6bc06ba3bbd P a97a6d07bc5e401781e977f7ef16f117 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23a2859f241b4df582667728cb114d0d" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 35813 } } peers { permanent_uuid: "a97a6d07bc5e401781e977f7ef16f117" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 33431 } } peers { permanent_uuid: "f2cd8925bb0a47c5adb02808b676d77e" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 38573 } }
I20260620 18:40:24.429965 7812 heartbeater.cc:503] Master 127.22.37.62:41813 was elected leader, sending a full tablet report...
I20260620 18:40:24.430686 7832 ts_tablet_manager.cc:1428] T dd6d00136e144074b235b6bc06ba3bbd P a97a6d07bc5e401781e977f7ef16f117: Time spent starting tablet: real 0.014s user 0.014s sys 0.001s
I20260620 18:40:24.433579 7831 raft_consensus.cc:348] T dd6d00136e144074b235b6bc06ba3bbd P 23a2859f241b4df582667728cb114d0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23a2859f241b4df582667728cb114d0d" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 35813 } } peers { permanent_uuid: "a97a6d07bc5e401781e977f7ef16f117" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 33431 } } peers { permanent_uuid: "f2cd8925bb0a47c5adb02808b676d77e" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 38573 } }
I20260620 18:40:24.434031 7831 raft_consensus.cc:374] T dd6d00136e144074b235b6bc06ba3bbd P 23a2859f241b4df582667728cb114d0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:40:24.434154 7831 raft_consensus.cc:729] T dd6d00136e144074b235b6bc06ba3bbd P 23a2859f241b4df582667728cb114d0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23a2859f241b4df582667728cb114d0d, State: Initialized, Role: FOLLOWER
I20260620 18:40:24.434772 7831 consensus_queue.cc:260] T dd6d00136e144074b235b6bc06ba3bbd P 23a2859f241b4df582667728cb114d0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23a2859f241b4df582667728cb114d0d" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 35813 } } peers { permanent_uuid: "a97a6d07bc5e401781e977f7ef16f117" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 33431 } } peers { permanent_uuid: "f2cd8925bb0a47c5adb02808b676d77e" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 38573 } }
I20260620 18:40:24.436362 7831 ts_tablet_manager.cc:1428] T dd6d00136e144074b235b6bc06ba3bbd P 23a2859f241b4df582667728cb114d0d: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
W20260620 18:40:24.547480 7546 tablet.cc:2369] T dd6d00136e144074b235b6bc06ba3bbd P 23a2859f241b4df582667728cb114d0d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:40:24.547595 7680 tablet.cc:2369] T dd6d00136e144074b235b6bc06ba3bbd P f2cd8925bb0a47c5adb02808b676d77e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:40:24.556368 7813 tablet.cc:2369] T dd6d00136e144074b235b6bc06ba3bbd P a97a6d07bc5e401781e977f7ef16f117: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:40:24.618567 7838 raft_consensus.cc:482] T dd6d00136e144074b235b6bc06ba3bbd P 23a2859f241b4df582667728cb114d0d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:40:24.618960 7838 raft_consensus.cc:504] T dd6d00136e144074b235b6bc06ba3bbd P 23a2859f241b4df582667728cb114d0d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23a2859f241b4df582667728cb114d0d" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 35813 } } peers { permanent_uuid: "a97a6d07bc5e401781e977f7ef16f117" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 33431 } } peers { permanent_uuid: "f2cd8925bb0a47c5adb02808b676d77e" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 38573 } }
I20260620 18:40:24.620755 7838 leader_election.cc:288] T dd6d00136e144074b235b6bc06ba3bbd P 23a2859f241b4df582667728cb114d0d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a97a6d07bc5e401781e977f7ef16f117 (127.22.37.3:33431), f2cd8925bb0a47c5adb02808b676d77e (127.22.37.2:38573)
I20260620 18:40:24.627501 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:24.621195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.1:45005 (local address 127.22.37.3:33431)
0620 18:40:24.621517 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:24.621524 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:24.622281 (+ 757us) server_negotiation.cc:407] Connection header received
0620 18:40:24.622506 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:24.622517 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:24.622575 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:24.622678 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:40:24.623262 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:24.623984 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:24.625844 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:24.626589 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:24.626747 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:24.626853 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:24.627125 (+ 272us) server_negotiation.cc:299] Negotiation successful
0620 18:40:24.627247 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154}
I20260620 18:40:24.627668 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:24.621312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.1:53553 (local address 127.22.37.2:38573)
0620 18:40:24.621530 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:24.621538 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:24.622428 (+ 890us) server_negotiation.cc:407] Connection header received
0620 18:40:24.622625 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:24.622635 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:24.622691 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:24.622787 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:40:24.623414 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:24.624095 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:24.625962 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:24.626738 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:24.626888 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:24.626992 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:24.627271 (+ 279us) server_negotiation.cc:299] Negotiation successful
0620 18:40:24.627397 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260620 18:40:24.627872 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:24.620951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.3:33431 (local address 127.22.37.1:45005)
0620 18:40:24.622171 (+ 1220us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:24.622188 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:40:24.622362 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:40:24.622692 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:40:24.622696 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:40:24.622719 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:40:24.623092 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:24.623130 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:24.624119 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:24.624126 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:24.625689 (+ 1563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:24.625701 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:24.626800 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:24.626806 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:24.626969 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:24.626994 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 18:40:24.627099 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 18:40:24.627230 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1090,"mutex_wait_us":36,"thread_start_us":890,"threads_started":1}
I20260620 18:40:24.628093 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:24.621203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.2:38573 (local address 127.22.37.1:53553)
0620 18:40:24.622315 (+ 1112us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:24.622328 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 18:40:24.622473 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:40:24.622805 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:40:24.622809 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:40:24.622844 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:40:24.623259 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:24.623267 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:24.624233 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:24.624239 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:24.625812 (+ 1573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:24.625823 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:24.626976 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:24.626982 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:24.627114 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:24.627141 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 18:40:24.627242 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 18:40:24.627360 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1001,"mutex_wait_us":42,"thread_start_us":781,"threads_started":1}
I20260620 18:40:24.628937 7767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd6d00136e144074b235b6bc06ba3bbd" candidate_uuid: "23a2859f241b4df582667728cb114d0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97a6d07bc5e401781e977f7ef16f117" is_pre_election: true
I20260620 18:40:24.629135 7634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd6d00136e144074b235b6bc06ba3bbd" candidate_uuid: "23a2859f241b4df582667728cb114d0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2cd8925bb0a47c5adb02808b676d77e" is_pre_election: true
I20260620 18:40:24.629568 7767 raft_consensus.cc:2454] T dd6d00136e144074b235b6bc06ba3bbd P a97a6d07bc5e401781e977f7ef16f117 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23a2859f241b4df582667728cb114d0d in term 0.
I20260620 18:40:24.629650 7634 raft_consensus.cc:2454] T dd6d00136e144074b235b6bc06ba3bbd P f2cd8925bb0a47c5adb02808b676d77e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23a2859f241b4df582667728cb114d0d in term 0.
I20260620 18:40:24.630340 7433 leader_election.cc:302] T dd6d00136e144074b235b6bc06ba3bbd P 23a2859f241b4df582667728cb114d0d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23a2859f241b4df582667728cb114d0d, f2cd8925bb0a47c5adb02808b676d77e; no voters:
I20260620 18:40:24.630752 7838 raft_consensus.cc:2781] T dd6d00136e144074b235b6bc06ba3bbd P 23a2859f241b4df582667728cb114d0d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:40:24.630916 7838 raft_consensus.cc:482] T dd6d00136e144074b235b6bc06ba3bbd P 23a2859f241b4df582667728cb114d0d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:40:24.631033 7838 raft_consensus.cc:3037] T dd6d00136e144074b235b6bc06ba3bbd P 23a2859f241b4df582667728cb114d0d [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:40:24.633143 7838 raft_consensus.cc:504] T dd6d00136e144074b235b6bc06ba3bbd P 23a2859f241b4df582667728cb114d0d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23a2859f241b4df582667728cb114d0d" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 35813 } } peers { permanent_uuid: "a97a6d07bc5e401781e977f7ef16f117" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 33431 } } peers { permanent_uuid: "f2cd8925bb0a47c5adb02808b676d77e" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 38573 } }
I20260620 18:40:24.633749 7838 leader_election.cc:288] T dd6d00136e144074b235b6bc06ba3bbd P 23a2859f241b4df582667728cb114d0d [CANDIDATE]: Term 1 election: Requested vote from peers a97a6d07bc5e401781e977f7ef16f117 (127.22.37.3:33431), f2cd8925bb0a47c5adb02808b676d77e (127.22.37.2:38573)
I20260620 18:40:24.634183 7634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd6d00136e144074b235b6bc06ba3bbd" candidate_uuid: "23a2859f241b4df582667728cb114d0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2cd8925bb0a47c5adb02808b676d77e"
I20260620 18:40:24.634375 7634 raft_consensus.cc:3037] T dd6d00136e144074b235b6bc06ba3bbd P f2cd8925bb0a47c5adb02808b676d77e [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:40:24.634521 7767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd6d00136e144074b235b6bc06ba3bbd" candidate_uuid: "23a2859f241b4df582667728cb114d0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97a6d07bc5e401781e977f7ef16f117"
I20260620 18:40:24.634721 7767 raft_consensus.cc:3037] T dd6d00136e144074b235b6bc06ba3bbd P a97a6d07bc5e401781e977f7ef16f117 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:40:24.636808 7634 raft_consensus.cc:2454] T dd6d00136e144074b235b6bc06ba3bbd P f2cd8925bb0a47c5adb02808b676d77e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23a2859f241b4df582667728cb114d0d in term 1.
I20260620 18:40:24.637307 7767 raft_consensus.cc:2454] T dd6d00136e144074b235b6bc06ba3bbd P a97a6d07bc5e401781e977f7ef16f117 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23a2859f241b4df582667728cb114d0d in term 1.
I20260620 18:40:24.637694 7433 leader_election.cc:302] T dd6d00136e144074b235b6bc06ba3bbd P 23a2859f241b4df582667728cb114d0d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23a2859f241b4df582667728cb114d0d, f2cd8925bb0a47c5adb02808b676d77e; no voters:
I20260620 18:40:24.637969 7838 raft_consensus.cc:2781] T dd6d00136e144074b235b6bc06ba3bbd P 23a2859f241b4df582667728cb114d0d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:40:24.638449 7838 raft_consensus.cc:686] T dd6d00136e144074b235b6bc06ba3bbd P 23a2859f241b4df582667728cb114d0d [term 1 LEADER]: Becoming Leader. State: Replica: 23a2859f241b4df582667728cb114d0d, State: Running, Role: LEADER
I20260620 18:40:24.638952 7838 consensus_queue.cc:237] T dd6d00136e144074b235b6bc06ba3bbd P 23a2859f241b4df582667728cb114d0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23a2859f241b4df582667728cb114d0d" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 35813 } } peers { permanent_uuid: "a97a6d07bc5e401781e977f7ef16f117" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 33431 } } peers { permanent_uuid: "f2cd8925bb0a47c5adb02808b676d77e" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 38573 } }
I20260620 18:40:24.643281 7356 catalog_manager.cc:5495] T dd6d00136e144074b235b6bc06ba3bbd P 23a2859f241b4df582667728cb114d0d reported cstate change: term changed from 0 to 1, leader changed from <none> to 23a2859f241b4df582667728cb114d0d (127.22.37.1). New cstate: current_term: 1 leader_uuid: "23a2859f241b4df582667728cb114d0d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23a2859f241b4df582667728cb114d0d" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 35813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a97a6d07bc5e401781e977f7ef16f117" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 33431 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f2cd8925bb0a47c5adb02808b676d77e" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 38573 } health_report { overall_health: UNKNOWN } } }
I20260620 18:40:24.682940 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:24.677061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43128 (local address 127.22.37.1:35813)
0620 18:40:24.677246 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:24.677251 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:24.677267 (+ 16us) server_negotiation.cc:407] Connection header received
0620 18:40:24.677515 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:24.677519 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:24.677593 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:24.677717 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:40:24.678120 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:24.678838 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:24.680838 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:24.681209 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:24.681338 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:24.681870 (+ 532us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:40:24.682367 (+ 497us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:40:24.682526 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:24.682585 (+ 59us) server_negotiation.cc:299] Negotiation successful
0620 18:40:24.682676 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 18:40:24.690939 7767 raft_consensus.cc:1261] T dd6d00136e144074b235b6bc06ba3bbd P a97a6d07bc5e401781e977f7ef16f117 [term 1 FOLLOWER]: Refusing update from remote peer 23a2859f241b4df582667728cb114d0d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:40:24.690937 7634 raft_consensus.cc:1261] T dd6d00136e144074b235b6bc06ba3bbd P f2cd8925bb0a47c5adb02808b676d77e [term 1 FOLLOWER]: Refusing update from remote peer 23a2859f241b4df582667728cb114d0d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:40:24.691926 7841 consensus_queue.cc:1035] T dd6d00136e144074b235b6bc06ba3bbd P 23a2859f241b4df582667728cb114d0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2cd8925bb0a47c5adb02808b676d77e" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 38573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:40:24.692278 7838 consensus_queue.cc:1035] T dd6d00136e144074b235b6bc06ba3bbd P 23a2859f241b4df582667728cb114d0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97a6d07bc5e401781e977f7ef16f117" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 33431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:40:24.716969 7846 mvcc.cc:205] Tried to move back new op lower bound from 7298993457924521984 to 7298993457724715008. Current Snapshot: MvccSnapshot[applied={T|T < 7298993457924521984}]
I20260620 18:40:24.723448 7355 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46438:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:40:24.723762 7355 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46438:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:40:24.727566 7355 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fb920b9ac1434a038cf12bccb256f3b3: Sending DeleteTablet for 3 replicas of tablet dd6d00136e144074b235b6bc06ba3bbd
I20260620 18:40:24.728853 7747 tablet_service.cc:1514] Processing DeleteTablet for tablet dd6d00136e144074b235b6bc06ba3bbd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:40:24 UTC) from {username='kudu', principal='kudu/127.22.37.62@KRBTEST.COM'} at 127.0.0.1:58738
I20260620 18:40:24.728899 7614 tablet_service.cc:1514] Processing DeleteTablet for tablet dd6d00136e144074b235b6bc06ba3bbd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:40:24 UTC) from {username='kudu', principal='kudu/127.22.37.62@KRBTEST.COM'} at 127.0.0.1:41562
I20260620 18:40:24.729105 7480 tablet_service.cc:1514] Processing DeleteTablet for tablet dd6d00136e144074b235b6bc06ba3bbd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:40:24 UTC) from {username='kudu', principal='kudu/127.22.37.62@KRBTEST.COM'} at 127.0.0.1:43112
I20260620 18:40:24.731637 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 7417
I20260620 18:40:24.740166 7856 tablet_replica.cc:330] T dd6d00136e144074b235b6bc06ba3bbd P a97a6d07bc5e401781e977f7ef16f117: stopping tablet replica
I20260620 18:40:24.740793 7856 raft_consensus.cc:2229] T dd6d00136e144074b235b6bc06ba3bbd P a97a6d07bc5e401781e977f7ef16f117 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:40:24.741497 7856 raft_consensus.cc:2258] T dd6d00136e144074b235b6bc06ba3bbd P a97a6d07bc5e401781e977f7ef16f117 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:40:24.743974 7856 ts_tablet_manager.cc:1905] T dd6d00136e144074b235b6bc06ba3bbd P a97a6d07bc5e401781e977f7ef16f117: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:40:24.751495 7856 ts_tablet_manager.cc:1918] T dd6d00136e144074b235b6bc06ba3bbd P a97a6d07bc5e401781e977f7ef16f117: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:40:24.751741 7856 log.cc:1196] T dd6d00136e144074b235b6bc06ba3bbd P a97a6d07bc5e401781e977f7ef16f117: Deleting WAL directory at /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980527431848-22676-0/minicluster-data/ts-2/wal/wals/dd6d00136e144074b235b6bc06ba3bbd
I20260620 18:40:24.752337 7856 ts_tablet_manager.cc:1939] T dd6d00136e144074b235b6bc06ba3bbd P a97a6d07bc5e401781e977f7ef16f117: Deleting consensus metadata
I20260620 18:40:24.754432 7342 catalog_manager.cc:4841] TS a97a6d07bc5e401781e977f7ef16f117 (127.22.37.3:33431): tablet dd6d00136e144074b235b6bc06ba3bbd (table test-table [id=5c29b8552dfe437e94cf8376762968de]) successfully deleted
I20260620 18:40:24.758463 7855 tablet_replica.cc:330] T dd6d00136e144074b235b6bc06ba3bbd P f2cd8925bb0a47c5adb02808b676d77e: stopping tablet replica
I20260620 18:40:24.759091 7855 raft_consensus.cc:2229] T dd6d00136e144074b235b6bc06ba3bbd P f2cd8925bb0a47c5adb02808b676d77e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:40:24.759704 7855 raft_consensus.cc:2258] T dd6d00136e144074b235b6bc06ba3bbd P f2cd8925bb0a47c5adb02808b676d77e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:40:24.760608 7857 tablet_replica.cc:330] T dd6d00136e144074b235b6bc06ba3bbd P 23a2859f241b4df582667728cb114d0d: stopping tablet replica
I20260620 18:40:24.761111 7857 raft_consensus.cc:2229] T dd6d00136e144074b235b6bc06ba3bbd P 23a2859f241b4df582667728cb114d0d [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:40:24.761883 7857 raft_consensus.cc:2258] T dd6d00136e144074b235b6bc06ba3bbd P 23a2859f241b4df582667728cb114d0d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:40:24.762331 7855 ts_tablet_manager.cc:1905] T dd6d00136e144074b235b6bc06ba3bbd P f2cd8925bb0a47c5adb02808b676d77e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:40:24.763761 7857 ts_tablet_manager.cc:1905] T dd6d00136e144074b235b6bc06ba3bbd P 23a2859f241b4df582667728cb114d0d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:40:24.767954 7855 ts_tablet_manager.cc:1918] T dd6d00136e144074b235b6bc06ba3bbd P f2cd8925bb0a47c5adb02808b676d77e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:40:24.768133 7855 log.cc:1196] T dd6d00136e144074b235b6bc06ba3bbd P f2cd8925bb0a47c5adb02808b676d77e: Deleting WAL directory at /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980527431848-22676-0/minicluster-data/ts-1/wal/wals/dd6d00136e144074b235b6bc06ba3bbd
I20260620 18:40:24.768639 7855 ts_tablet_manager.cc:1939] T dd6d00136e144074b235b6bc06ba3bbd P f2cd8925bb0a47c5adb02808b676d77e: Deleting consensus metadata
I20260620 18:40:24.770268 7340 catalog_manager.cc:4841] TS f2cd8925bb0a47c5adb02808b676d77e (127.22.37.2:38573): tablet dd6d00136e144074b235b6bc06ba3bbd (table test-table [id=5c29b8552dfe437e94cf8376762968de]) successfully deleted
I20260620 18:40:24.772045 7857 ts_tablet_manager.cc:1918] T dd6d00136e144074b235b6bc06ba3bbd P 23a2859f241b4df582667728cb114d0d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:40:24.772219 7857 log.cc:1196] T dd6d00136e144074b235b6bc06ba3bbd P 23a2859f241b4df582667728cb114d0d: Deleting WAL directory at /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980527431848-22676-0/minicluster-data/ts-0/wal/wals/dd6d00136e144074b235b6bc06ba3bbd
Jun 20 18:40:24 dist-test-slave-5lgp krb5kdc[7311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.37.1@KRBTEST.COM
I20260620 18:40:24.772774 7857 ts_tablet_manager.cc:1939] T dd6d00136e144074b235b6bc06ba3bbd P 23a2859f241b4df582667728cb114d0d: Deleting consensus metadata
I20260620 18:40:24.774605 7341 catalog_manager.cc:4841] TS 23a2859f241b4df582667728cb114d0d (127.22.37.1:35813): tablet dd6d00136e144074b235b6bc06ba3bbd (table test-table [id=5c29b8552dfe437e94cf8376762968de]) successfully deleted
I20260620 18:40:24.779968 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:24.732544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43136 (local address 127.22.37.1:35813)
0620 18:40:24.732734 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:24.732740 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:24.756944 (+ 24204us) server_negotiation.cc:407] Connection header received
0620 18:40:24.757250 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:24.757260 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:24.757405 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:24.757501 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:40:24.759866 (+ 2365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:24.761244 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:24.763340 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:24.764038 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:24.764147 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:24.773831 (+ 9684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:40:24.773847 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:40:24.773861 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:40:24.773903 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:40:24.777033 (+ 3130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:40:24.777840 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:40:24.777847 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:40:24.777849 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:40:24.777991 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:40:24.778398 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:40:24.778403 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:40:24.778405 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:40:24.778751 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:40:24.779091 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:24.779387 (+ 296us) server_negotiation.cc:299] Negotiation successful
0620 18:40:24.779631 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 18:40:24.780646 7540 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:40:24.884006 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 7417
I20260620 18:40:24.902523 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 7551
Jun 20 18:40:24 dist-test-slave-5lgp krb5kdc[7311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.37.2@KRBTEST.COM
I20260620 18:40:24.918241 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:24.903370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41570 (local address 127.22.37.2:38573)
0620 18:40:24.903574 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:24.903582 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:24.903607 (+ 25us) server_negotiation.cc:407] Connection header received
0620 18:40:24.903912 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:24.903916 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:24.903982 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:24.904096 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:40:24.905462 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:24.906264 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:24.907572 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:24.908022 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:24.908151 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:24.912469 (+ 4318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:40:24.912487 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:40:24.912503 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:40:24.912543 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:40:24.915377 (+ 2834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:40:24.916025 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:40:24.916033 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:40:24.916035 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:40:24.916166 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:40:24.916529 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:40:24.916536 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:40:24.916539 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:40:24.916892 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:40:24.917234 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:24.917687 (+ 453us) server_negotiation.cc:299] Negotiation successful
0620 18:40:24.917909 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 18:40:24.918900 7674 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:40:25.008843 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 7551
I20260620 18:40:25.027037 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 7684
Jun 20 18:40:25 dist-test-slave-5lgp krb5kdc[7311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.37.3@KRBTEST.COM
I20260620 18:40:25.043095 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:25.027871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58740 (local address 127.22.37.3:33431)
0620 18:40:25.028038 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:25.028043 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:25.028094 (+ 51us) server_negotiation.cc:407] Connection header received
0620 18:40:25.028262 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:25.028266 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:25.028341 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:25.028427 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:40:25.030033 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:25.030776 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:25.032200 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:25.032618 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:25.032731 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:25.037536 (+ 4805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:40:25.037551 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:40:25.037568 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:40:25.037604 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:40:25.040249 (+ 2645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:40:25.040959 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:40:25.040966 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:40:25.040969 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:40:25.041119 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:40:25.041578 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:40:25.041582 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:40:25.041584 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:40:25.041943 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:40:25.042165 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:25.042620 (+ 455us) server_negotiation.cc:299] Negotiation successful
0620 18:40:25.042804 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 18:40:25.043821 7807 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:40:25.133473 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 7684
I20260620 18:40:25.153477 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 7324
I20260620 18:40:25.154680 7386 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:40:25.266544 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 7324
2026-06-20T18:40:25Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4393 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-20T18:40:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:40:25Z Disabled control of system clock
I20260620 18:40:25.347987 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980527431848-22676-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980527431848-22676-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980527431848-22676-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.62:37905
--webserver_interface=127.22.37.62
--webserver_port=0
--builtin_ntp_servers=127.22.37.20:40247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.37.62:37905
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 18:40:25.673574 7866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:40:25.673861 7866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:40:25.674053 7866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:40:25.681345 7866 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:40:25.681509 7866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:40:25.681581 7866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:40:25.681687 7866 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:40:25.681777 7866 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:40:25.691062 7866 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:40247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980527431848-22676-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980527431848-22676-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.37.62:37905
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.62:37905
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.37.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980527431848-22676-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:40:25.692456 7866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:40:25.694311 7866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:40:25.706007 7871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:40:25.706030 7874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:40:25.705991 7872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:40:25.706878 7866 server_base.cc:1034] running on GCE node
I20260620 18:40:25.707593 7866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:40:25.709013 7866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:40:25.710216 7866 hybrid_clock.cc:648] HybridClock initialized: now 1781980825710165 us; error 43 us; skew 500 ppm
I20260620 18:40:25.710567 7866 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:40:25.712733 7866 webserver.cc:460] Webserver started at http://127.22.37.62:45807/ using document root <none> and password file <none>
I20260620 18:40:25.713459 7866 fs_manager.cc:359] Metadata directory not provided
I20260620 18:40:25.713573 7866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:40:25.713898 7866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:40:25.716610 7866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980527431848-22676-0/minicluster-data/master-0/data/instance:
uuid: "c862aa4d12e2445285b4591e5a74d836"
format_stamp: "Formatted at 2026-06-20 18:40:25 on dist-test-slave-5lgp"
server_key: "7f4c779cf93d3023eac05052354c5805"
server_key_iv: "120372bd76ed02df89a74a31c17ec48e"
server_key_version: "encryptionkey@0"
I20260620 18:40:25.717481 7866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980527431848-22676-0/minicluster-data/master-0/wal/instance:
uuid: "c862aa4d12e2445285b4591e5a74d836"
format_stamp: "Formatted at 2026-06-20 18:40:25 on dist-test-slave-5lgp"
server_key: "7f4c779cf93d3023eac05052354c5805"
server_key_iv: "120372bd76ed02df89a74a31c17ec48e"
server_key_version: "encryptionkey@0"
I20260620 18:40:25.722272 7866 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 18:40:25.725644 7880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:40:25.726863 7866 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 18:40:25.727035 7866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980527431848-22676-0/minicluster-data/master-0/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980527431848-22676-0/minicluster-data/master-0/wal
uuid: "c862aa4d12e2445285b4591e5a74d836"
format_stamp: "Formatted at 2026-06-20 18:40:25 on dist-test-slave-5lgp"
server_key: "7f4c779cf93d3023eac05052354c5805"
server_key_iv: "120372bd76ed02df89a74a31c17ec48e"
server_key_version: "encryptionkey@0"
I20260620 18:40:25.727232 7866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980527431848-22676-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980527431848-22676-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980527431848-22676-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:40:25.744204 7866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:40:25.745026 7866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:40:25.745317 7866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:40:25.763880 7866 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.62:37905
I20260620 18:40:25.763912 7931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.62:37905 every 8 connection(s)
I20260620 18:40:25.765539 7866 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
I20260620 18:40:25.769301 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 7866
I20260620 18:40:25.769520 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980527431848-22676-0/minicluster-data/master-0/wal/instance
I20260620 18:40:25.769550 7932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:40:25.770254 22676 external_mini_cluster.cc:1428] Setting key 55665db6d3171a09c0ea7a781f66722f
I20260620 18:40:25.782852 7932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c862aa4d12e2445285b4591e5a74d836: Bootstrap starting.
I20260620 18:40:25.783690 7935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:25.772214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55604 (local address 127.22.37.62:37905)
0620 18:40:25.773251 (+ 1037us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:25.773266 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:25.773310 (+ 44us) server_negotiation.cc:407] Connection header received
0620 18:40:25.775016 (+ 1706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:25.775039 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:25.775533 (+ 494us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:25.775967 (+ 434us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:40:25.776562 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:25.777941 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:25.779311 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:25.780015 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:25.780238 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:25.781453 (+ 1215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:40:25.781463 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:40:25.781482 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:40:25.782021 (+ 539us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:40:25.782098 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:40:25.782119 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:40:25.782240 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:25.782456 (+ 216us) server_negotiation.cc:299] Negotiation successful
0620 18:40:25.782757 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":375,"threads_started":1}
I20260620 18:40:25.786042 7932 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c862aa4d12e2445285b4591e5a74d836: Neither blocks nor log segments found. Creating new log.
I20260620 18:40:25.787125 7932 log.cc:824] T 00000000000000000000000000000000 P c862aa4d12e2445285b4591e5a74d836: Log is configured to *not* fsync() on all Append() calls
I20260620 18:40:25.790066 7932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c862aa4d12e2445285b4591e5a74d836: No bootstrap required, opened a new log
I20260620 18:40:25.798219 7932 raft_consensus.cc:348] T 00000000000000000000000000000000 P c862aa4d12e2445285b4591e5a74d836 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c862aa4d12e2445285b4591e5a74d836" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 37905 } }
I20260620 18:40:25.798619 7932 raft_consensus.cc:374] T 00000000000000000000000000000000 P c862aa4d12e2445285b4591e5a74d836 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:40:25.798740 7932 raft_consensus.cc:729] T 00000000000000000000000000000000 P c862aa4d12e2445285b4591e5a74d836 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c862aa4d12e2445285b4591e5a74d836, State: Initialized, Role: FOLLOWER
I20260620 18:40:25.799293 7932 consensus_queue.cc:260] T 00000000000000000000000000000000 P c862aa4d12e2445285b4591e5a74d836 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c862aa4d12e2445285b4591e5a74d836" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 37905 } }
I20260620 18:40:25.799544 7932 raft_consensus.cc:388] T 00000000000000000000000000000000 P c862aa4d12e2445285b4591e5a74d836 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:40:25.799691 7932 raft_consensus.cc:482] T 00000000000000000000000000000000 P c862aa4d12e2445285b4591e5a74d836 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:40:25.799876 7932 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c862aa4d12e2445285b4591e5a74d836 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:40:25.802214 7932 raft_consensus.cc:504] T 00000000000000000000000000000000 P c862aa4d12e2445285b4591e5a74d836 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c862aa4d12e2445285b4591e5a74d836" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 37905 } }
I20260620 18:40:25.802749 7932 leader_election.cc:302] T 00000000000000000000000000000000 P c862aa4d12e2445285b4591e5a74d836 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c862aa4d12e2445285b4591e5a74d836; no voters:
I20260620 18:40:25.803335 7932 leader_election.cc:288] T 00000000000000000000000000000000 P c862aa4d12e2445285b4591e5a74d836 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:40:25.803457 7937 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c862aa4d12e2445285b4591e5a74d836 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:40:25.804186 7937 raft_consensus.cc:686] T 00000000000000000000000000000000 P c862aa4d12e2445285b4591e5a74d836 [term 1 LEADER]: Becoming Leader. State: Replica: c862aa4d12e2445285b4591e5a74d836, State: Running, Role: LEADER
I20260620 18:40:25.804656 7937 consensus_queue.cc:237] T 00000000000000000000000000000000 P c862aa4d12e2445285b4591e5a74d836 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c862aa4d12e2445285b4591e5a74d836" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 37905 } }
I20260620 18:40:25.805440 7932 sys_catalog.cc:564] T 00000000000000000000000000000000 P c862aa4d12e2445285b4591e5a74d836 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:40:25.807464 7939 sys_catalog.cc:455] T 00000000000000000000000000000000 P c862aa4d12e2445285b4591e5a74d836 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c862aa4d12e2445285b4591e5a74d836. Latest consensus state: current_term: 1 leader_uuid: "c862aa4d12e2445285b4591e5a74d836" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c862aa4d12e2445285b4591e5a74d836" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 37905 } } }
I20260620 18:40:25.807495 7938 sys_catalog.cc:455] T 00000000000000000000000000000000 P c862aa4d12e2445285b4591e5a74d836 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c862aa4d12e2445285b4591e5a74d836" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c862aa4d12e2445285b4591e5a74d836" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 37905 } } }
I20260620 18:40:25.807897 7939 sys_catalog.cc:458] T 00000000000000000000000000000000 P c862aa4d12e2445285b4591e5a74d836 [sys.catalog]: This master's current role is: LEADER
I20260620 18:40:25.807991 7938 sys_catalog.cc:458] T 00000000000000000000000000000000 P c862aa4d12e2445285b4591e5a74d836 [sys.catalog]: This master's current role is: LEADER
I20260620 18:40:25.810688 7942 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:40:25.815371 7942 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:40:25.823946 7942 catalog_manager.cc:1347] Generated new cluster ID: f315961f1c884bed89b17c8cd6ab454e
I20260620 18:40:25.824081 7942 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:40:25.844628 7942 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:40:25.846042 7942 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:40:25.855620 7942 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c862aa4d12e2445285b4591e5a74d836: Generated new TSK 0
I20260620 18:40:25.856491 7942 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:40:25.864588 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980527431848-22676-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980527431848-22676-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980527431848-22676-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980527431848-22676-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.37.1:0
--local_ip_for_outbound_sockets=127.22.37.1
--webserver_interface=127.22.37.1
--webserver_port=0
--tserver_master_addrs=127.22.37.62:37905
--builtin_ntp_servers=127.22.37.20:40247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 18:40:26.203356 7956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:40:26.203631 7956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:40:26.203723 7956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:40:26.211407 7956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:40:26.211524 7956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:40:26.211700 7956 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.37.1
I20260620 18:40:26.221169 7956 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:40247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980527431848-22676-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980527431848-22676-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.37.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980527431848-22676-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.37.1
--webserver_port=0
--tserver_master_addrs=127.22.37.62:37905
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.37.1
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980527431848-22676-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:40:26.222499 7956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:40:26.224306 7956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:40:26.237977 7961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:40:26.238008 7964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:40:26.238047 7962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:40:26.239681 7956 server_base.cc:1034] running on GCE node
I20260620 18:40:26.240298 7956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:40:26.241276 7956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:40:26.242507 7956 hybrid_clock.cc:648] HybridClock initialized: now 1781980826242445 us; error 58 us; skew 500 ppm
I20260620 18:40:26.242828 7956 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:40:26.245086 7956 webserver.cc:460] Webserver started at http://127.22.37.1:41177/ using document root <none> and password file <none>
I20260620 18:40:26.245846 7956 fs_manager.cc:359] Metadata directory not provided
I20260620 18:40:26.245980 7956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:40:26.246297 7956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:40:26.249174 7956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980527431848-22676-0/minicluster-data/ts-0/data/instance:
uuid: "0cb7e90fc1a646bebdd1a16234845062"
format_stamp: "Formatted at 2026-06-20 18:40:26 on dist-test-slave-5lgp"
server_key: "61e76fe8f4a7c059eb5e7060f6d11efc"
server_key_iv: "bcbcd13a48dec2ea4cfbf627a302a830"
server_key_version: "encryptionkey@0"
I20260620 18:40:26.250015 7956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980527431848-22676-0/minicluster-data/ts-0/wal/instance:
uuid: "0cb7e90fc1a646bebdd1a16234845062"
format_stamp: "Formatted at 2026-06-20 18:40:26 on dist-test-slave-5lgp"
server_key: "61e76fe8f4a7c059eb5e7060f6d11efc"
server_key_iv: "bcbcd13a48dec2ea4cfbf627a302a830"
server_key_version: "encryptionkey@0"
I20260620 18:40:26.255014 7956 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:40:26.258544 7970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:40:26.259832 7956 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:40:26.260002 7956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980527431848-22676-0/minicluster-data/ts-0/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980527431848-22676-0/minicluster-data/ts-0/wal
uuid: "0cb7e90fc1a646bebdd1a16234845062"
format_stamp: "Formatted at 2026-06-20 18:40:26 on dist-test-slave-5lgp"
server_key: "61e76fe8f4a7c059eb5e7060f6d11efc"
server_key_iv: "bcbcd13a48dec2ea4cfbf627a302a830"
server_key_version: "encryptionkey@0"
I20260620 18:40:26.260215 7956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980527431848-22676-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980527431848-22676-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980527431848-22676-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:40:26.307035 7956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:40:26.307971 7956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:40:26.308239 7956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:40:26.309351 7956 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:40:26.311034 7956 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:40:26.311163 7956 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:40:26.311285 7956 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:40:26.311369 7956 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:40:26.344399 7956 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.1:44775
I20260620 18:40:26.344457 8082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.1:44775 every 8 connection(s)
I20260620 18:40:26.345745 7956 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980527431848-22676-0/minicluster-data/ts-0/data/info.pb
I20260620 18:40:26.348824 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 7956
I20260620 18:40:26.349014 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980527431848-22676-0/minicluster-data/ts-0/wal/instance
I20260620 18:40:26.349655 22676 external_mini_cluster.cc:1428] Setting key 4bcd45c2de8dea73c1745a4adcfb34d6
I20260620 18:40:26.352758 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980527431848-22676-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980527431848-22676-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980527431848-22676-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980527431848-22676-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.37.2:0
--local_ip_for_outbound_sockets=127.22.37.2
--webserver_interface=127.22.37.2
--webserver_port=0
--tserver_master_addrs=127.22.37.62:37905
--builtin_ntp_servers=127.22.37.20:40247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 18:40:26.360899 8085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:26.348356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.1:34715 (local address 127.22.37.62:37905)
0620 18:40:26.348875 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:26.348882 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:26.350359 (+ 1477us) server_negotiation.cc:407] Connection header received
0620 18:40:26.352647 (+ 2288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:26.352652 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:26.352714 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:26.352796 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:40:26.354306 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:26.355258 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:26.357821 (+ 2563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:26.358238 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:26.358340 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:26.360005 (+ 1665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:40:26.360010 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:40:26.360013 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:40:26.360033 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:40:26.360071 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:40:26.360083 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:40:26.360156 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:26.360526 (+ 370us) server_negotiation.cc:299] Negotiation successful
0620 18:40:26.360610 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":272,"threads_started":1}
I20260620 18:40:26.362397 8086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:26.348992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:37905 (local address 127.22.37.1:34715)
0620 18:40:26.350166 (+ 1174us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:26.350225 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 18:40:26.352322 (+ 2097us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:40:26.353091 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:40:26.353105 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:40:26.353230 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:40:26.354130 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:26.354159 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:26.355433 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:26.355440 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:26.357640 (+ 2200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:26.357657 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:26.358468 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:26.358478 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:26.358820 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:26.359627 (+ 807us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:40:26.359633 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:40:26.359690 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:40:26.359794 (+ 104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:40:26.360176 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:40:26.360191 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:40:26.360202 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 18:40:26.360385 (+ 183us) client_negotiation.cc:240] Negotiation successful
0620 18:40:26.360877 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":331,"threads_started":1}
I20260620 18:40:26.364060 8083 heartbeater.cc:347] Connected to a master server at 127.22.37.62:37905
I20260620 18:40:26.364379 8083 heartbeater.cc:464] Registering TS with master...
I20260620 18:40:26.365183 8083 heartbeater.cc:511] Master 127.22.37.62:37905 requested a full tablet report, sending...
I20260620 18:40:26.367256 7897 ts_manager.cc:194] Registered new tserver with Master: 0cb7e90fc1a646bebdd1a16234845062 (127.22.37.1:44775)
I20260620 18:40:26.368692 7897 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.37.1:34715
W20260620 18:40:26.675345 8087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:40:26.675607 8087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:40:26.675760 8087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:40:26.682785 8087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:40:26.682879 8087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:40:26.683048 8087 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.37.2
I20260620 18:40:26.691720 8087 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:40247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980527431848-22676-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980527431848-22676-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.37.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980527431848-22676-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.37.2
--webserver_port=0
--tserver_master_addrs=127.22.37.62:37905
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.37.2
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980527431848-22676-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:40:26.692979 8087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:40:26.694708 8087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:40:26.707276 8092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:40:26.707310 8093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:40:26.707346 8095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:40:26.708534 8087 server_base.cc:1034] running on GCE node
I20260620 18:40:26.709165 8087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:40:26.710189 8087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:40:26.711395 8087 hybrid_clock.cc:648] HybridClock initialized: now 1781980826711358 us; error 30 us; skew 500 ppm
I20260620 18:40:26.711714 8087 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:40:26.714059 8087 webserver.cc:460] Webserver started at http://127.22.37.2:42693/ using document root <none> and password file <none>
I20260620 18:40:26.714721 8087 fs_manager.cc:359] Metadata directory not provided
I20260620 18:40:26.714833 8087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:40:26.715145 8087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:40:26.718055 8087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980527431848-22676-0/minicluster-data/ts-1/data/instance:
uuid: "ff4d02a820204fff9a394e93bf35344f"
format_stamp: "Formatted at 2026-06-20 18:40:26 on dist-test-slave-5lgp"
server_key: "e1838f2a69244b9153aed9361fd94a08"
server_key_iv: "7b817ee08322d682effcf97626309d36"
server_key_version: "encryptionkey@0"
I20260620 18:40:26.718820 8087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980527431848-22676-0/minicluster-data/ts-1/wal/instance:
uuid: "ff4d02a820204fff9a394e93bf35344f"
format_stamp: "Formatted at 2026-06-20 18:40:26 on dist-test-slave-5lgp"
server_key: "e1838f2a69244b9153aed9361fd94a08"
server_key_iv: "7b817ee08322d682effcf97626309d36"
server_key_version: "encryptionkey@0"
I20260620 18:40:26.723820 8087 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:40:26.727190 8101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:40:26.728448 8087 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 18:40:26.728618 8087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980527431848-22676-0/minicluster-data/ts-1/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980527431848-22676-0/minicluster-data/ts-1/wal
uuid: "ff4d02a820204fff9a394e93bf35344f"
format_stamp: "Formatted at 2026-06-20 18:40:26 on dist-test-slave-5lgp"
server_key: "e1838f2a69244b9153aed9361fd94a08"
server_key_iv: "7b817ee08322d682effcf97626309d36"
server_key_version: "encryptionkey@0"
I20260620 18:40:26.728883 8087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980527431848-22676-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980527431848-22676-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980527431848-22676-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:40:26.748252 8087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:40:26.748993 8087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:40:26.749269 8087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:40:26.750216 8087 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:40:26.751554 8087 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:40:26.751673 8087 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:40:26.751801 8087 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:40:26.751907 8087 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:40:26.787199 8087 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.2:46705
I20260620 18:40:26.787284 8213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.2:46705 every 8 connection(s)
I20260620 18:40:26.788477 8087 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980527431848-22676-0/minicluster-data/ts-1/data/info.pb
I20260620 18:40:26.789036 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 8087
I20260620 18:40:26.789240 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980527431848-22676-0/minicluster-data/ts-1/wal/instance
I20260620 18:40:26.789839 22676 external_mini_cluster.cc:1428] Setting key cba9a500430e61bb7984f31c35f36022
I20260620 18:40:26.793040 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980527431848-22676-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980527431848-22676-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980527431848-22676-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980527431848-22676-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.37.3:0
--local_ip_for_outbound_sockets=127.22.37.3
--webserver_interface=127.22.37.3
--webserver_port=0
--tserver_master_addrs=127.22.37.62:37905
--builtin_ntp_servers=127.22.37.20:40247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 18:40:26.803779 8085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:26.791319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.2:46647 (local address 127.22.37.62:37905)
0620 18:40:26.791514 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:26.791520 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:26.793300 (+ 1780us) server_negotiation.cc:407] Connection header received
0620 18:40:26.796064 (+ 2764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:26.796069 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:26.796146 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:26.796223 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:40:26.797719 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:26.798879 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:26.800987 (+ 2108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:26.801443 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:26.801582 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:26.802978 (+ 1396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:40:26.802985 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:40:26.802989 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:40:26.803016 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:40:26.803047 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:40:26.803059 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:40:26.803155 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:26.803378 (+ 223us) server_negotiation.cc:299] Negotiation successful
0620 18:40:26.803473 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 18:40:26.804795 8216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:26.791900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:37905 (local address 127.22.37.2:46647)
0620 18:40:26.793017 (+ 1117us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:26.793068 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 18:40:26.795772 (+ 2704us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:40:26.796486 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:40:26.796505 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:40:26.796668 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:40:26.797523 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:26.797552 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:26.799086 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:26.799092 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:26.800840 (+ 1748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:26.800853 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:26.801672 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:26.801679 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:26.801917 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:26.802652 (+ 735us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:40:26.802655 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:40:26.802699 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:40:26.802782 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:40:26.803158 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:40:26.803171 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:40:26.803180 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 18:40:26.803369 (+ 189us) client_negotiation.cc:240] Negotiation successful
0620 18:40:26.803755 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":355,"threads_started":1}
I20260620 18:40:26.806221 8214 heartbeater.cc:347] Connected to a master server at 127.22.37.62:37905
I20260620 18:40:26.806536 8214 heartbeater.cc:464] Registering TS with master...
I20260620 18:40:26.807335 8214 heartbeater.cc:511] Master 127.22.37.62:37905 requested a full tablet report, sending...
I20260620 18:40:26.808925 7897 ts_manager.cc:194] Registered new tserver with Master: ff4d02a820204fff9a394e93bf35344f (127.22.37.2:46705)
I20260620 18:40:26.810081 7897 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.37.2:46647
W20260620 18:40:27.126046 8217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:40:27.126381 8217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:40:27.126544 8217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:40:27.134276 8217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:40:27.134416 8217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:40:27.134586 8217 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.37.3
I20260620 18:40:27.145021 8217 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:40247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980527431848-22676-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980527431848-22676-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.37.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980527431848-22676-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.37.3
--webserver_port=0
--tserver_master_addrs=127.22.37.62:37905
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.37.3
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980527431848-22676-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:40:27.146685 8217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:40:27.148486 8217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:40:27.161630 8222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:40:27.161664 8223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:40:27.161861 8225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:40:27.162880 8217 server_base.cc:1034] running on GCE node
I20260620 18:40:27.163486 8217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:40:27.164490 8217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:40:27.165730 8217 hybrid_clock.cc:648] HybridClock initialized: now 1781980827165696 us; error 45 us; skew 500 ppm
I20260620 18:40:27.166055 8217 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:40:27.168278 8217 webserver.cc:460] Webserver started at http://127.22.37.3:36953/ using document root <none> and password file <none>
I20260620 18:40:27.169000 8217 fs_manager.cc:359] Metadata directory not provided
I20260620 18:40:27.169149 8217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:40:27.169534 8217 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:40:27.172387 8217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980527431848-22676-0/minicluster-data/ts-2/data/instance:
uuid: "e371b6895743481faa20a700f37b848a"
format_stamp: "Formatted at 2026-06-20 18:40:27 on dist-test-slave-5lgp"
server_key: "70fdafec756c442597f945401269fe04"
server_key_iv: "b56024d046a5afba6adc73fa0cc624b7"
server_key_version: "encryptionkey@0"
I20260620 18:40:27.173188 8217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980527431848-22676-0/minicluster-data/ts-2/wal/instance:
uuid: "e371b6895743481faa20a700f37b848a"
format_stamp: "Formatted at 2026-06-20 18:40:27 on dist-test-slave-5lgp"
server_key: "70fdafec756c442597f945401269fe04"
server_key_iv: "b56024d046a5afba6adc73fa0cc624b7"
server_key_version: "encryptionkey@0"
I20260620 18:40:27.178315 8217 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 18:40:27.181704 8231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:40:27.183023 8217 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 18:40:27.183203 8217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980527431848-22676-0/minicluster-data/ts-2/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980527431848-22676-0/minicluster-data/ts-2/wal
uuid: "e371b6895743481faa20a700f37b848a"
format_stamp: "Formatted at 2026-06-20 18:40:27 on dist-test-slave-5lgp"
server_key: "70fdafec756c442597f945401269fe04"
server_key_iv: "b56024d046a5afba6adc73fa0cc624b7"
server_key_version: "encryptionkey@0"
I20260620 18:40:27.183414 8217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980527431848-22676-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980527431848-22676-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980527431848-22676-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:40:27.213096 8217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:40:27.214005 8217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:40:27.214293 8217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:40:27.215337 8217 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:40:27.216807 8217 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:40:27.216933 8217 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:40:27.217069 8217 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:40:27.217157 8217 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:40:27.251734 8217 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.3:42811
I20260620 18:40:27.251791 8343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.3:42811 every 8 connection(s)
I20260620 18:40:27.253058 8217 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980527431848-22676-0/minicluster-data/ts-2/data/info.pb
I20260620 18:40:27.260959 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 8217
I20260620 18:40:27.261163 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980527431848-22676-0/minicluster-data/ts-2/wal/instance
I20260620 18:40:27.261688 22676 external_mini_cluster.cc:1428] Setting key 5ad785c65f466e0fbdd36f6a3843d42e
I20260620 18:40:27.266007 8085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:27.255658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.3:60185 (local address 127.22.37.62:37905)
0620 18:40:27.255841 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:27.255847 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:27.257219 (+ 1372us) server_negotiation.cc:407] Connection header received
0620 18:40:27.259057 (+ 1838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:27.259063 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:27.259154 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:27.259267 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:40:27.260317 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:27.261120 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:27.263005 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:27.263573 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:27.263716 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:27.265155 (+ 1439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:40:27.265161 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:40:27.265166 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:40:27.265193 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:40:27.265223 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:40:27.265235 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:40:27.265325 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:27.265578 (+ 253us) server_negotiation.cc:299] Negotiation successful
0620 18:40:27.265680 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 18:40:27.266997 8346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:27.256177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:37905 (local address 127.22.37.3:60185)
0620 18:40:27.257046 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:27.257086 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 18:40:27.258789 (+ 1703us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:40:27.259501 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:40:27.259509 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:40:27.259596 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:40:27.260151 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:27.260167 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:27.261294 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:27.261298 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:27.262862 (+ 1564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:27.262876 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:27.263806 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:27.263813 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:27.264047 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:27.264814 (+ 767us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:40:27.264818 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:40:27.264873 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:40:27.264967 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:40:27.265338 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:40:27.265351 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:40:27.265359 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 18:40:27.265571 (+ 212us) client_negotiation.cc:240] Negotiation successful
0620 18:40:27.265961 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":265,"threads_started":1}
I20260620 18:40:27.268276 8344 heartbeater.cc:347] Connected to a master server at 127.22.37.62:37905
I20260620 18:40:27.268576 8344 heartbeater.cc:464] Registering TS with master...
I20260620 18:40:27.269289 8344 heartbeater.cc:511] Master 127.22.37.62:37905 requested a full tablet report, sending...
I20260620 18:40:27.270673 7897 ts_manager.cc:194] Registered new tserver with Master: e371b6895743481faa20a700f37b848a (127.22.37.3:42811)
I20260620 18:40:27.271456 7897 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.37.3:60185
I20260620 18:40:27.276856 22676 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:40:27.286267 8085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:27.280914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55610 (local address 127.22.37.62:37905)
0620 18:40:27.281105 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:27.281109 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:27.281570 (+ 461us) server_negotiation.cc:407] Connection header received
0620 18:40:27.281701 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:27.281704 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:27.281772 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:27.281851 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:40:27.282292 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:27.282949 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:27.284242 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:27.284617 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:27.284704 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:27.285672 (+ 968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:40:27.285676 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:40:27.285679 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:40:27.285695 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:40:27.285723 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:40:27.285734 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:40:27.285801 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:27.285913 (+ 112us) server_negotiation.cc:299] Negotiation successful
0620 18:40:27.285979 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 18:40:27.289952 7897 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55610:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:40:27.296749 7897 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:40:27.332268 8358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:27.309360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58846 (local address 127.22.37.2:46705)
0620 18:40:27.316296 (+ 6936us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:27.316310 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:27.322493 (+ 6183us) server_negotiation.cc:407] Connection header received
0620 18:40:27.323139 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:27.323148 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:27.323524 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:27.323758 (+ 234us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:40:27.324952 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:27.326523 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:27.329142 (+ 2619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:27.330719 (+ 1577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:27.330847 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:27.331123 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:27.331726 (+ 603us) server_negotiation.cc:299] Negotiation successful
0620 18:40:27.331907 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6667,"thread_start_us":6529,"threads_started":1}
I20260620 18:40:27.332261 8360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:27.309738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.2:46705 (local address 127.0.0.1:58846)
0620 18:40:27.322317 (+ 12579us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:27.322344 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 18:40:27.322892 (+ 548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:40:27.324154 (+ 1262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:40:27.324165 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:40:27.324206 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:40:27.324709 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:27.324724 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:27.326691 (+ 1967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:27.326696 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:27.328931 (+ 2235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:27.328945 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:27.331260 (+ 2315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:27.331266 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:27.331494 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:27.331530 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 18:40:27.331675 (+ 145us) client_negotiation.cc:240] Negotiation successful
0620 18:40:27.331882 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12370,"thread_start_us":12769,"threads_started":1}
I20260620 18:40:27.333320 8357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:27.309041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42734 (local address 127.22.37.3:42811)
0620 18:40:27.325310 (+ 16269us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:27.325322 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:27.325354 (+ 32us) server_negotiation.cc:407] Connection header received
0620 18:40:27.325481 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:27.325490 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:27.325698 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:27.325909 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:40:27.326561 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:27.328129 (+ 1568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:27.330443 (+ 2314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:27.331838 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:27.332453 (+ 615us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:27.332720 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:27.332794 (+ 74us) server_negotiation.cc:299] Negotiation successful
0620 18:40:27.332965 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16098,"thread_start_us":15959,"threads_started":1}
I20260620 18:40:27.333451 8356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:27.308557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.3:42811 (local address 127.0.0.1:42734)
0620 18:40:27.309309 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:27.309352 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 18:40:27.309550 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:40:27.325902 (+ 16352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:40:27.325907 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:40:27.325932 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:40:27.326310 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:27.326320 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:27.328295 (+ 1975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:27.328300 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:27.330226 (+ 1926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:27.330242 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:27.332055 (+ 1813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:27.332062 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:27.332265 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:27.332299 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 18:40:27.332354 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 18:40:27.333069 (+ 715us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"mutex_wait_us":544,"thread_start_us":362,"threads_started":1}
I20260620 18:40:27.333669 8359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:27.309743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.1:44775 (local address 127.0.0.1:48122)
0620 18:40:27.322811 (+ 13068us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:27.322835 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 18:40:27.322988 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:40:27.325523 (+ 2535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:40:27.325527 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:40:27.325554 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:40:27.325848 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:27.325858 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:27.327710 (+ 1852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:27.327715 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:27.329596 (+ 1881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:27.329615 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:27.331806 (+ 2191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:27.331812 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:27.332107 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:27.332149 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 18:40:27.332297 (+ 148us) client_negotiation.cc:240] Negotiation successful
0620 18:40:27.333199 (+ 902us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12902,"mutex_wait_us":254,"thread_start_us":13314,"threads_started":1}
I20260620 18:40:27.334300 8361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:27.310097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48122 (local address 127.22.37.1:44775)
0620 18:40:27.324847 (+ 14750us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:27.324856 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:27.324888 (+ 32us) server_negotiation.cc:407] Connection header received
0620 18:40:27.325004 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:27.325013 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:27.325272 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:27.325537 (+ 265us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:40:27.326039 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:27.327508 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:27.329804 (+ 2296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:27.331294 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:27.331428 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:27.331670 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:27.333000 (+ 1330us) server_negotiation.cc:299] Negotiation successful
0620 18:40:27.333974 (+ 974us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14613,"thread_start_us":14498,"threads_started":1}
I20260620 18:40:27.336489 8149 tablet_service.cc:1467] Processing CreateTablet for tablet 3281d3314bc44efbb700b5f3c0fa9745 (DEFAULT_TABLE table=test-table [id=dab0be47990e4a7cb62fcf79ed87902d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:40:27.337793 8149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3281d3314bc44efbb700b5f3c0fa9745. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:40:27.337419 8018 tablet_service.cc:1467] Processing CreateTablet for tablet 3281d3314bc44efbb700b5f3c0fa9745 (DEFAULT_TABLE table=test-table [id=dab0be47990e4a7cb62fcf79ed87902d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:40:27.338881 8018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3281d3314bc44efbb700b5f3c0fa9745. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:40:27.340054 8279 tablet_service.cc:1467] Processing CreateTablet for tablet 3281d3314bc44efbb700b5f3c0fa9745 (DEFAULT_TABLE table=test-table [id=dab0be47990e4a7cb62fcf79ed87902d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:40:27.341490 8279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3281d3314bc44efbb700b5f3c0fa9745. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:40:27.357276 8362 tablet_bootstrap.cc:492] T 3281d3314bc44efbb700b5f3c0fa9745 P ff4d02a820204fff9a394e93bf35344f: Bootstrap starting.
I20260620 18:40:27.357921 8363 tablet_bootstrap.cc:492] T 3281d3314bc44efbb700b5f3c0fa9745 P e371b6895743481faa20a700f37b848a: Bootstrap starting.
I20260620 18:40:27.361857 8363 tablet_bootstrap.cc:654] T 3281d3314bc44efbb700b5f3c0fa9745 P e371b6895743481faa20a700f37b848a: Neither blocks nor log segments found. Creating new log.
I20260620 18:40:27.362217 8364 tablet_bootstrap.cc:492] T 3281d3314bc44efbb700b5f3c0fa9745 P 0cb7e90fc1a646bebdd1a16234845062: Bootstrap starting.
I20260620 18:40:27.362704 8362 tablet_bootstrap.cc:654] T 3281d3314bc44efbb700b5f3c0fa9745 P ff4d02a820204fff9a394e93bf35344f: Neither blocks nor log segments found. Creating new log.
I20260620 18:40:27.363209 8363 log.cc:824] T 3281d3314bc44efbb700b5f3c0fa9745 P e371b6895743481faa20a700f37b848a: Log is configured to *not* fsync() on all Append() calls
I20260620 18:40:27.363847 8362 log.cc:824] T 3281d3314bc44efbb700b5f3c0fa9745 P ff4d02a820204fff9a394e93bf35344f: Log is configured to *not* fsync() on all Append() calls
I20260620 18:40:27.366084 8364 tablet_bootstrap.cc:654] T 3281d3314bc44efbb700b5f3c0fa9745 P 0cb7e90fc1a646bebdd1a16234845062: Neither blocks nor log segments found. Creating new log.
I20260620 18:40:27.366706 8363 tablet_bootstrap.cc:492] T 3281d3314bc44efbb700b5f3c0fa9745 P e371b6895743481faa20a700f37b848a: No bootstrap required, opened a new log
I20260620 18:40:27.366974 8363 ts_tablet_manager.cc:1397] T 3281d3314bc44efbb700b5f3c0fa9745 P e371b6895743481faa20a700f37b848a: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260620 18:40:27.367018 8362 tablet_bootstrap.cc:492] T 3281d3314bc44efbb700b5f3c0fa9745 P ff4d02a820204fff9a394e93bf35344f: No bootstrap required, opened a new log
I20260620 18:40:27.367280 8362 ts_tablet_manager.cc:1397] T 3281d3314bc44efbb700b5f3c0fa9745 P ff4d02a820204fff9a394e93bf35344f: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20260620 18:40:27.367395 8364 log.cc:824] T 3281d3314bc44efbb700b5f3c0fa9745 P 0cb7e90fc1a646bebdd1a16234845062: Log is configured to *not* fsync() on all Append() calls
I20260620 18:40:27.370267 8364 tablet_bootstrap.cc:492] T 3281d3314bc44efbb700b5f3c0fa9745 P 0cb7e90fc1a646bebdd1a16234845062: No bootstrap required, opened a new log
I20260620 18:40:27.370517 8364 ts_tablet_manager.cc:1397] T 3281d3314bc44efbb700b5f3c0fa9745 P 0cb7e90fc1a646bebdd1a16234845062: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.005s
I20260620 18:40:27.371505 8083 heartbeater.cc:503] Master 127.22.37.62:37905 was elected leader, sending a full tablet report...
I20260620 18:40:27.376027 8363 raft_consensus.cc:348] T 3281d3314bc44efbb700b5f3c0fa9745 P e371b6895743481faa20a700f37b848a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e371b6895743481faa20a700f37b848a" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 42811 } } peers { permanent_uuid: "0cb7e90fc1a646bebdd1a16234845062" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44775 } } peers { permanent_uuid: "ff4d02a820204fff9a394e93bf35344f" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 46705 } }
I20260620 18:40:27.376616 8363 raft_consensus.cc:374] T 3281d3314bc44efbb700b5f3c0fa9745 P e371b6895743481faa20a700f37b848a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:40:27.376765 8363 raft_consensus.cc:729] T 3281d3314bc44efbb700b5f3c0fa9745 P e371b6895743481faa20a700f37b848a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e371b6895743481faa20a700f37b848a, State: Initialized, Role: FOLLOWER
I20260620 18:40:27.377596 8363 consensus_queue.cc:260] T 3281d3314bc44efbb700b5f3c0fa9745 P e371b6895743481faa20a700f37b848a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e371b6895743481faa20a700f37b848a" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 42811 } } peers { permanent_uuid: "0cb7e90fc1a646bebdd1a16234845062" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44775 } } peers { permanent_uuid: "ff4d02a820204fff9a394e93bf35344f" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 46705 } }
I20260620 18:40:27.377641 8362 raft_consensus.cc:348] T 3281d3314bc44efbb700b5f3c0fa9745 P ff4d02a820204fff9a394e93bf35344f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e371b6895743481faa20a700f37b848a" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 42811 } } peers { permanent_uuid: "0cb7e90fc1a646bebdd1a16234845062" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44775 } } peers { permanent_uuid: "ff4d02a820204fff9a394e93bf35344f" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 46705 } }
I20260620 18:40:27.378094 8362 raft_consensus.cc:374] T 3281d3314bc44efbb700b5f3c0fa9745 P ff4d02a820204fff9a394e93bf35344f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:40:27.378194 8362 raft_consensus.cc:729] T 3281d3314bc44efbb700b5f3c0fa9745 P ff4d02a820204fff9a394e93bf35344f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff4d02a820204fff9a394e93bf35344f, State: Initialized, Role: FOLLOWER
I20260620 18:40:27.378772 8362 consensus_queue.cc:260] T 3281d3314bc44efbb700b5f3c0fa9745 P ff4d02a820204fff9a394e93bf35344f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e371b6895743481faa20a700f37b848a" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 42811 } } peers { permanent_uuid: "0cb7e90fc1a646bebdd1a16234845062" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44775 } } peers { permanent_uuid: "ff4d02a820204fff9a394e93bf35344f" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 46705 } }
I20260620 18:40:27.379981 8214 heartbeater.cc:503] Master 127.22.37.62:37905 was elected leader, sending a full tablet report...
I20260620 18:40:27.381021 8364 raft_consensus.cc:348] T 3281d3314bc44efbb700b5f3c0fa9745 P 0cb7e90fc1a646bebdd1a16234845062 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e371b6895743481faa20a700f37b848a" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 42811 } } peers { permanent_uuid: "0cb7e90fc1a646bebdd1a16234845062" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44775 } } peers { permanent_uuid: "ff4d02a820204fff9a394e93bf35344f" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 46705 } }
I20260620 18:40:27.381512 8364 raft_consensus.cc:374] T 3281d3314bc44efbb700b5f3c0fa9745 P 0cb7e90fc1a646bebdd1a16234845062 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:40:27.381662 8364 raft_consensus.cc:729] T 3281d3314bc44efbb700b5f3c0fa9745 P 0cb7e90fc1a646bebdd1a16234845062 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cb7e90fc1a646bebdd1a16234845062, State: Initialized, Role: FOLLOWER
I20260620 18:40:27.381969 8362 ts_tablet_manager.cc:1428] T 3281d3314bc44efbb700b5f3c0fa9745 P ff4d02a820204fff9a394e93bf35344f: Time spent starting tablet: real 0.015s user 0.011s sys 0.003s
I20260620 18:40:27.382411 8364 consensus_queue.cc:260] T 3281d3314bc44efbb700b5f3c0fa9745 P 0cb7e90fc1a646bebdd1a16234845062 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e371b6895743481faa20a700f37b848a" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 42811 } } peers { permanent_uuid: "0cb7e90fc1a646bebdd1a16234845062" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44775 } } peers { permanent_uuid: "ff4d02a820204fff9a394e93bf35344f" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 46705 } }
I20260620 18:40:27.382841 8344 heartbeater.cc:503] Master 127.22.37.62:37905 was elected leader, sending a full tablet report...
I20260620 18:40:27.384552 8363 ts_tablet_manager.cc:1428] T 3281d3314bc44efbb700b5f3c0fa9745 P e371b6895743481faa20a700f37b848a: Time spent starting tablet: real 0.017s user 0.014s sys 0.003s
I20260620 18:40:27.386459 8364 ts_tablet_manager.cc:1428] T 3281d3314bc44efbb700b5f3c0fa9745 P 0cb7e90fc1a646bebdd1a16234845062: Time spent starting tablet: real 0.016s user 0.013s sys 0.003s
W20260620 18:40:27.504843 8345 tablet.cc:2369] T 3281d3314bc44efbb700b5f3c0fa9745 P e371b6895743481faa20a700f37b848a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:40:27.540652 8215 tablet.cc:2369] T 3281d3314bc44efbb700b5f3c0fa9745 P ff4d02a820204fff9a394e93bf35344f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:40:27.598359 8084 tablet.cc:2369] T 3281d3314bc44efbb700b5f3c0fa9745 P 0cb7e90fc1a646bebdd1a16234845062: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:40:27.670040 8369 raft_consensus.cc:482] T 3281d3314bc44efbb700b5f3c0fa9745 P ff4d02a820204fff9a394e93bf35344f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:40:27.670403 8369 raft_consensus.cc:504] T 3281d3314bc44efbb700b5f3c0fa9745 P ff4d02a820204fff9a394e93bf35344f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e371b6895743481faa20a700f37b848a" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 42811 } } peers { permanent_uuid: "0cb7e90fc1a646bebdd1a16234845062" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44775 } } peers { permanent_uuid: "ff4d02a820204fff9a394e93bf35344f" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 46705 } }
I20260620 18:40:27.672220 8369 leader_election.cc:288] T 3281d3314bc44efbb700b5f3c0fa9745 P ff4d02a820204fff9a394e93bf35344f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e371b6895743481faa20a700f37b848a (127.22.37.3:42811), 0cb7e90fc1a646bebdd1a16234845062 (127.22.37.1:44775)
I20260620 18:40:27.679199 8357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:27.672717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.2:60519 (local address 127.22.37.3:42811)
0620 18:40:27.672929 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:27.672935 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:27.673175 (+ 240us) server_negotiation.cc:407] Connection header received
0620 18:40:27.673435 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:27.673443 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:27.673537 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:27.673635 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:40:27.674334 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:27.675262 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:27.677296 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:27.678220 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:27.678316 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:27.678458 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:27.678803 (+ 345us) server_negotiation.cc:299] Negotiation successful
0620 18:40:27.678937 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 18:40:27.679298 8371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:27.672421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.3:42811 (local address 127.22.37.2:60519)
0620 18:40:27.673049 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:27.673065 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:40:27.673256 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:40:27.673748 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:40:27.673756 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:40:27.673799 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:40:27.674187 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:27.674200 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:27.675395 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:27.675400 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:27.677101 (+ 1701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:27.677113 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:27.678404 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:27.678411 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:27.678634 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:27.678670 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 18:40:27.678766 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 18:40:27.678903 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"mutex_wait_us":61,"thread_start_us":342,"threads_started":1}
I20260620 18:40:27.679598 8361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:27.673520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.2:43283 (local address 127.22.37.1:44775)
0620 18:40:27.673731 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:27.673737 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:27.673931 (+ 194us) server_negotiation.cc:407] Connection header received
0620 18:40:27.674086 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:27.674091 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:27.674185 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:27.674289 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:40:27.674862 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:27.675706 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:27.677601 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:27.678662 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:27.678808 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:27.678893 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:27.679222 (+ 329us) server_negotiation.cc:299] Negotiation successful
0620 18:40:27.679371 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 18:40:27.679981 8372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:27.673190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.1:44775 (local address 127.22.37.2:43283)
0620 18:40:27.673829 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:27.673842 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 18:40:27.673958 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:40:27.674328 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:40:27.674332 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:40:27.674355 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:40:27.674724 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:27.674735 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:27.675851 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:27.675855 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:27.677406 (+ 1551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:27.677420 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:27.678869 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:27.678875 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:27.679048 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:27.679082 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 18:40:27.679172 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 18:40:27.679520 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"mutex_wait_us":56,"thread_start_us":341,"threads_started":1}
I20260620 18:40:27.680511 8299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3281d3314bc44efbb700b5f3c0fa9745" candidate_uuid: "ff4d02a820204fff9a394e93bf35344f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e371b6895743481faa20a700f37b848a" is_pre_election: true
I20260620 18:40:27.681056 8299 raft_consensus.cc:2454] T 3281d3314bc44efbb700b5f3c0fa9745 P e371b6895743481faa20a700f37b848a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff4d02a820204fff9a394e93bf35344f in term 0.
I20260620 18:40:27.680992 8038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3281d3314bc44efbb700b5f3c0fa9745" candidate_uuid: "ff4d02a820204fff9a394e93bf35344f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cb7e90fc1a646bebdd1a16234845062" is_pre_election: true
I20260620 18:40:27.681643 8038 raft_consensus.cc:2454] T 3281d3314bc44efbb700b5f3c0fa9745 P 0cb7e90fc1a646bebdd1a16234845062 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff4d02a820204fff9a394e93bf35344f in term 0.
I20260620 18:40:27.681749 8103 leader_election.cc:302] T 3281d3314bc44efbb700b5f3c0fa9745 P ff4d02a820204fff9a394e93bf35344f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e371b6895743481faa20a700f37b848a, ff4d02a820204fff9a394e93bf35344f; no voters:
I20260620 18:40:27.682207 8369 raft_consensus.cc:2781] T 3281d3314bc44efbb700b5f3c0fa9745 P ff4d02a820204fff9a394e93bf35344f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:40:27.682358 8369 raft_consensus.cc:482] T 3281d3314bc44efbb700b5f3c0fa9745 P ff4d02a820204fff9a394e93bf35344f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:40:27.682467 8369 raft_consensus.cc:3037] T 3281d3314bc44efbb700b5f3c0fa9745 P ff4d02a820204fff9a394e93bf35344f [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:40:27.684660 8369 raft_consensus.cc:504] T 3281d3314bc44efbb700b5f3c0fa9745 P ff4d02a820204fff9a394e93bf35344f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e371b6895743481faa20a700f37b848a" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 42811 } } peers { permanent_uuid: "0cb7e90fc1a646bebdd1a16234845062" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44775 } } peers { permanent_uuid: "ff4d02a820204fff9a394e93bf35344f" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 46705 } }
I20260620 18:40:27.685334 8369 leader_election.cc:288] T 3281d3314bc44efbb700b5f3c0fa9745 P ff4d02a820204fff9a394e93bf35344f [CANDIDATE]: Term 1 election: Requested vote from peers e371b6895743481faa20a700f37b848a (127.22.37.3:42811), 0cb7e90fc1a646bebdd1a16234845062 (127.22.37.1:44775)
I20260620 18:40:27.685840 8038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3281d3314bc44efbb700b5f3c0fa9745" candidate_uuid: "ff4d02a820204fff9a394e93bf35344f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cb7e90fc1a646bebdd1a16234845062"
I20260620 18:40:27.685837 8299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3281d3314bc44efbb700b5f3c0fa9745" candidate_uuid: "ff4d02a820204fff9a394e93bf35344f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e371b6895743481faa20a700f37b848a"
I20260620 18:40:27.686061 8299 raft_consensus.cc:3037] T 3281d3314bc44efbb700b5f3c0fa9745 P e371b6895743481faa20a700f37b848a [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:40:27.686092 8038 raft_consensus.cc:3037] T 3281d3314bc44efbb700b5f3c0fa9745 P 0cb7e90fc1a646bebdd1a16234845062 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:40:27.688479 8299 raft_consensus.cc:2454] T 3281d3314bc44efbb700b5f3c0fa9745 P e371b6895743481faa20a700f37b848a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff4d02a820204fff9a394e93bf35344f in term 1.
I20260620 18:40:27.688508 8038 raft_consensus.cc:2454] T 3281d3314bc44efbb700b5f3c0fa9745 P 0cb7e90fc1a646bebdd1a16234845062 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff4d02a820204fff9a394e93bf35344f in term 1.
I20260620 18:40:27.688907 8103 leader_election.cc:302] T 3281d3314bc44efbb700b5f3c0fa9745 P ff4d02a820204fff9a394e93bf35344f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e371b6895743481faa20a700f37b848a, ff4d02a820204fff9a394e93bf35344f; no voters:
I20260620 18:40:27.689225 8369 raft_consensus.cc:2781] T 3281d3314bc44efbb700b5f3c0fa9745 P ff4d02a820204fff9a394e93bf35344f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:40:27.689668 8369 raft_consensus.cc:686] T 3281d3314bc44efbb700b5f3c0fa9745 P ff4d02a820204fff9a394e93bf35344f [term 1 LEADER]: Becoming Leader. State: Replica: ff4d02a820204fff9a394e93bf35344f, State: Running, Role: LEADER
I20260620 18:40:27.690173 8369 consensus_queue.cc:237] T 3281d3314bc44efbb700b5f3c0fa9745 P ff4d02a820204fff9a394e93bf35344f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e371b6895743481faa20a700f37b848a" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 42811 } } peers { permanent_uuid: "0cb7e90fc1a646bebdd1a16234845062" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44775 } } peers { permanent_uuid: "ff4d02a820204fff9a394e93bf35344f" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 46705 } }
I20260620 18:40:27.694897 7897 catalog_manager.cc:5495] T 3281d3314bc44efbb700b5f3c0fa9745 P ff4d02a820204fff9a394e93bf35344f reported cstate change: term changed from 0 to 1, leader changed from <none> to ff4d02a820204fff9a394e93bf35344f (127.22.37.2). New cstate: current_term: 1 leader_uuid: "ff4d02a820204fff9a394e93bf35344f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e371b6895743481faa20a700f37b848a" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 42811 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cb7e90fc1a646bebdd1a16234845062" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff4d02a820204fff9a394e93bf35344f" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 46705 } health_report { overall_health: HEALTHY } } }
I20260620 18:40:27.793982 8358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:27.787275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58858 (local address 127.22.37.2:46705)
0620 18:40:27.787498 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:27.787509 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:27.787698 (+ 189us) server_negotiation.cc:407] Connection header received
0620 18:40:27.787858 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:27.787863 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:27.787978 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:27.788122 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:40:27.788583 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:27.789744 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:27.791230 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:27.791958 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:27.792082 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:27.792560 (+ 478us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:40:27.793268 (+ 708us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:40:27.793434 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:27.793606 (+ 172us) server_negotiation.cc:299] Negotiation successful
0620 18:40:27.793689 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 18:40:27.804459 8038 raft_consensus.cc:1261] T 3281d3314bc44efbb700b5f3c0fa9745 P 0cb7e90fc1a646bebdd1a16234845062 [term 1 FOLLOWER]: Refusing update from remote peer ff4d02a820204fff9a394e93bf35344f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:40:27.804561 8299 raft_consensus.cc:1261] T 3281d3314bc44efbb700b5f3c0fa9745 P e371b6895743481faa20a700f37b848a [term 1 FOLLOWER]: Refusing update from remote peer ff4d02a820204fff9a394e93bf35344f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:40:27.805632 8373 consensus_queue.cc:1035] T 3281d3314bc44efbb700b5f3c0fa9745 P ff4d02a820204fff9a394e93bf35344f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e371b6895743481faa20a700f37b848a" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 42811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:40:27.806322 8373 consensus_queue.cc:1035] T 3281d3314bc44efbb700b5f3c0fa9745 P ff4d02a820204fff9a394e93bf35344f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb7e90fc1a646bebdd1a16234845062" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:40:27.821105 8377 mvcc.cc:205] Tried to move back new op lower bound from 7298993470675066880 to 7298993470223245312. Current Snapshot: MvccSnapshot[applied={T|T < 7298993470675066880}]
I20260620 18:40:27.821830 8378 mvcc.cc:205] Tried to move back new op lower bound from 7298993470675066880 to 7298993470223245312. Current Snapshot: MvccSnapshot[applied={T|T < 7298993470675066880}]
I20260620 18:40:27.822124 8379 mvcc.cc:205] Tried to move back new op lower bound from 7298993470675066880 to 7298993470223245312. Current Snapshot: MvccSnapshot[applied={T|T < 7298993470675066880}]
I20260620 18:40:27.838122 7897 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:40:27.838374 7897 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:40:27.841886 7897 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c862aa4d12e2445285b4591e5a74d836: Sending DeleteTablet for 3 replicas of tablet 3281d3314bc44efbb700b5f3c0fa9745
I20260620 18:40:27.843099 8279 tablet_service.cc:1514] Processing DeleteTablet for tablet 3281d3314bc44efbb700b5f3c0fa9745 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:40:27 UTC) from {username='slave'} at 127.0.0.1:42734
I20260620 18:40:27.843175 8018 tablet_service.cc:1514] Processing DeleteTablet for tablet 3281d3314bc44efbb700b5f3c0fa9745 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:40:27 UTC) from {username='slave'} at 127.0.0.1:48122
I20260620 18:40:27.843853 8149 tablet_service.cc:1514] Processing DeleteTablet for tablet 3281d3314bc44efbb700b5f3c0fa9745 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:40:27 UTC) from {username='slave'} at 127.0.0.1:58846
I20260620 18:40:27.844633 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 7956
I20260620 18:40:27.864490 8386 tablet_replica.cc:330] T 3281d3314bc44efbb700b5f3c0fa9745 P e371b6895743481faa20a700f37b848a: stopping tablet replica
I20260620 18:40:27.867259 8361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:27.847483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48124 (local address 127.22.37.1:44775)
0620 18:40:27.847721 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:27.847730 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:27.856357 (+ 8627us) server_negotiation.cc:407] Connection header received
0620 18:40:27.856888 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:27.856897 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:27.857054 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:27.857143 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:40:27.857933 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:27.859391 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:27.861896 (+ 2505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:27.862737 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:27.862852 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:27.865174 (+ 2322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:40:27.865190 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:40:27.865215 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:40:27.866195 (+ 980us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:40:27.866269 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:40:27.866298 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:40:27.866418 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:27.866741 (+ 323us) server_negotiation.cc:299] Negotiation successful
0620 18:40:27.866865 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 18:40:27.868130 8078 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:40:27.868628 8386 raft_consensus.cc:2229] T 3281d3314bc44efbb700b5f3c0fa9745 P e371b6895743481faa20a700f37b848a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:40:27.869436 8386 raft_consensus.cc:2258] T 3281d3314bc44efbb700b5f3c0fa9745 P e371b6895743481faa20a700f37b848a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:40:27.870371 8388 tablet_replica.cc:330] T 3281d3314bc44efbb700b5f3c0fa9745 P ff4d02a820204fff9a394e93bf35344f: stopping tablet replica
I20260620 18:40:27.870792 8388 raft_consensus.cc:2229] T 3281d3314bc44efbb700b5f3c0fa9745 P ff4d02a820204fff9a394e93bf35344f [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:40:27.871482 8388 raft_consensus.cc:2258] T 3281d3314bc44efbb700b5f3c0fa9745 P ff4d02a820204fff9a394e93bf35344f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:40:27.872354 8386 ts_tablet_manager.cc:1905] T 3281d3314bc44efbb700b5f3c0fa9745 P e371b6895743481faa20a700f37b848a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:40:27.873216 8388 ts_tablet_manager.cc:1905] T 3281d3314bc44efbb700b5f3c0fa9745 P ff4d02a820204fff9a394e93bf35344f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:40:27.880369 8386 ts_tablet_manager.cc:1918] T 3281d3314bc44efbb700b5f3c0fa9745 P e371b6895743481faa20a700f37b848a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:40:27.880724 8386 log.cc:1196] T 3281d3314bc44efbb700b5f3c0fa9745 P e371b6895743481faa20a700f37b848a: Deleting WAL directory at /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980527431848-22676-0/minicluster-data/ts-2/wal/wals/3281d3314bc44efbb700b5f3c0fa9745
I20260620 18:40:27.881438 8386 ts_tablet_manager.cc:1939] T 3281d3314bc44efbb700b5f3c0fa9745 P e371b6895743481faa20a700f37b848a: Deleting consensus metadata
I20260620 18:40:27.883560 7882 catalog_manager.cc:4841] TS e371b6895743481faa20a700f37b848a (127.22.37.3:42811): tablet 3281d3314bc44efbb700b5f3c0fa9745 (table test-table [id=dab0be47990e4a7cb62fcf79ed87902d]) successfully deleted
I20260620 18:40:27.885442 8388 ts_tablet_manager.cc:1918] T 3281d3314bc44efbb700b5f3c0fa9745 P ff4d02a820204fff9a394e93bf35344f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:40:27.885639 8388 log.cc:1196] T 3281d3314bc44efbb700b5f3c0fa9745 P ff4d02a820204fff9a394e93bf35344f: Deleting WAL directory at /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980527431848-22676-0/minicluster-data/ts-1/wal/wals/3281d3314bc44efbb700b5f3c0fa9745
I20260620 18:40:27.886292 8388 ts_tablet_manager.cc:1939] T 3281d3314bc44efbb700b5f3c0fa9745 P ff4d02a820204fff9a394e93bf35344f: Deleting consensus metadata
I20260620 18:40:27.888646 7883 catalog_manager.cc:4841] TS ff4d02a820204fff9a394e93bf35344f (127.22.37.2:46705): tablet 3281d3314bc44efbb700b5f3c0fa9745 (table test-table [id=dab0be47990e4a7cb62fcf79ed87902d]) successfully deleted
I20260620 18:40:27.977967 8387 tablet_replica.cc:330] T 3281d3314bc44efbb700b5f3c0fa9745 P 0cb7e90fc1a646bebdd1a16234845062: stopping tablet replica
I20260620 18:40:27.978497 8387 raft_consensus.cc:2229] T 3281d3314bc44efbb700b5f3c0fa9745 P 0cb7e90fc1a646bebdd1a16234845062 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:40:27.979000 8387 raft_consensus.cc:2258] T 3281d3314bc44efbb700b5f3c0fa9745 P 0cb7e90fc1a646bebdd1a16234845062 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:40:27.980262 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 7956
W20260620 18:40:27.998905 7882 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.37.1:44775 (error 108)
I20260620 18:40:28.000281 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 8087
I20260620 18:40:28.001206 8392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:28.000237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.1:44775 (local address 127.0.0.1:48134)
0620 18:40:28.000815 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:28.000970 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.1:44775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":286,"threads_started":1}
W20260620 18:40:28.001760 7882 catalog_manager.cc:4584] TS 0cb7e90fc1a646bebdd1a16234845062 (127.22.37.1:44775): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3281d3314bc44efbb700b5f3c0fa9745: Network error: Client connection negotiation failed: client connection to 127.22.37.1:44775: connect: Connection refused (error 111)
I20260620 18:40:28.007944 8358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:28.001435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58872 (local address 127.22.37.2:46705)
0620 18:40:28.001651 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:28.001659 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:28.001685 (+ 26us) server_negotiation.cc:407] Connection header received
0620 18:40:28.001741 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:28.001745 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:28.001820 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:28.001946 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:40:28.002393 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:28.003162 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:28.004488 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:28.004955 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:28.005094 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:28.006329 (+ 1235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:40:28.006340 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:40:28.006359 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:40:28.007131 (+ 772us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:40:28.007185 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:40:28.007205 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:40:28.007331 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:28.007491 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 18:40:28.007609 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 18:40:28.008620 8209 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:40:28.055783 8392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:28.055246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.1:44775 (local address 127.0.0.1:48150)
0620 18:40:28.055522 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:28.055625 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.1:44775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260620 18:40:28.056903 8392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:28.056542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.1:44775 (local address 127.0.0.1:48152)
0620 18:40:28.056700 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:28.056767 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.1:44775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260620 18:40:28.117530 8392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:28.116941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.1:44775 (local address 127.0.0.1:48166)
0620 18:40:28.117179 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:28.117284 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.1:44775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260620 18:40:28.118676 8392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:28.118319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.1:44775 (local address 127.0.0.1:48168)
0620 18:40:28.118460 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:28.118540 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.1:44775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 18:40:28.128227 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 8087
I20260620 18:40:28.147011 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 8217
I20260620 18:40:28.153728 8357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:28.147796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42748 (local address 127.22.37.3:42811)
0620 18:40:28.147977 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:28.147981 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:28.148016 (+ 35us) server_negotiation.cc:407] Connection header received
0620 18:40:28.148194 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:28.148198 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:28.148278 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:28.148361 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:40:28.148712 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:28.149498 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:28.150767 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:28.151166 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:28.151254 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:28.152279 (+ 1025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:40:28.152287 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:40:28.152301 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:40:28.152846 (+ 545us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:40:28.152887 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:40:28.152905 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:40:28.152989 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:28.153163 (+ 174us) server_negotiation.cc:299] Negotiation successful
0620 18:40:28.153250 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 18:40:28.154425 8339 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:40:28.199306 8392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:28.198711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.1:44775 (local address 127.0.0.1:48184)
0620 18:40:28.198971 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:28.199122 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.1:44775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260620 18:40:28.200476 8392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:28.200009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.1:44775 (local address 127.0.0.1:48200)
0620 18:40:28.200213 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:28.200297 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.1:44775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260620 18:40:28.243294 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 8217
I20260620 18:40:28.260190 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 7866
I20260620 18:40:28.261334 7927 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:40:28.372329 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 7866
2026-06-20T18:40:28Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (3103 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980527431848-22676-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:40:28 dist-test-slave-5lgp krb5kdc[8398](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:40:28 dist-test-slave-5lgp krb5kdc[8398](info): set up 2 sockets
Jun 20 18:40:28 dist-test-slave-5lgp krb5kdc[8398](info): commencing operation
krb5kdc: starting...
W20260620 18:40:29.485565 22676 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980527431848-22676-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980527431848-22676-0/krb5kdc/test-user.keytab.
Jun 20 18:40:29 dist-test-slave-5lgp krb5kdc[8398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980829, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:40:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:40:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.37.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.37.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.37.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab.
Entry for principal kudu/127.22.37.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab.
I20260620 18:40:29.730885 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980527431848-22676-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980527431848-22676-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980527431848-22676-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.62:38937
--webserver_interface=127.22.37.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab
--principal=kudu/127.22.37.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:42013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.37.62:38937
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:40:30.073596 8411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:40:30.074010 8411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:40:30.074191 8411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:40:30.081796 8411 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:40:30.081990 8411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:40:30.082118 8411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:40:30.082229 8411 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:40:30.082321 8411 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:40:30.092818 8411 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:42013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980527431848-22676-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980527431848-22676-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.37.62:38937
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab
--principal=kudu/127.22.37.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.62:38937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.37.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980527431848-22676-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:40:30.094429 8411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:40:30.095854 8411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:40:30.108747 8416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:40:30.108793 8419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:40:30.108846 8417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:40:30.110131 8411 server_base.cc:1034] running on GCE node
I20260620 18:40:30.110993 8411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:40:30.112497 8411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:40:30.113754 8411 hybrid_clock.cc:648] HybridClock initialized: now 1781980830113693 us; error 41 us; skew 500 ppm
I20260620 18:40:30.114164 8411 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:40:30 dist-test-slave-5lgp krb5kdc[8398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980830, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:40:30.121461 8411 init.cc:377] Logged in from keytab as kudu/127.22.37.62@KRBTEST.COM (short username kudu)
I20260620 18:40:30.123157 8411 webserver.cc:460] Webserver started at http://127.22.37.62:38169/ using document root <none> and password file <none>
I20260620 18:40:30.123893 8411 fs_manager.cc:359] Metadata directory not provided
I20260620 18:40:30.124054 8411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:40:30.124383 8411 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:40:30.127562 8411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980527431848-22676-0/minicluster-data/master-0/data/instance:
uuid: "fdbe0c57133444f6b3379a5cf82d6bfc"
format_stamp: "Formatted at 2026-06-20 18:40:30 on dist-test-slave-5lgp"
server_key: "c329ddad5ff3e1114b7510aff6a6dd86"
server_key_iv: "ab3ad24c0bed022ea6893583427c34d1"
server_key_version: "encryptionkey@0"
I20260620 18:40:30.128351 8411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980527431848-22676-0/minicluster-data/master-0/wal/instance:
uuid: "fdbe0c57133444f6b3379a5cf82d6bfc"
format_stamp: "Formatted at 2026-06-20 18:40:30 on dist-test-slave-5lgp"
server_key: "c329ddad5ff3e1114b7510aff6a6dd86"
server_key_iv: "ab3ad24c0bed022ea6893583427c34d1"
server_key_version: "encryptionkey@0"
I20260620 18:40:30.133910 8411 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 18:40:30.137725 8426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:40:30.139154 8411 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:40:30.139370 8411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980527431848-22676-0/minicluster-data/master-0/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980527431848-22676-0/minicluster-data/master-0/wal
uuid: "fdbe0c57133444f6b3379a5cf82d6bfc"
format_stamp: "Formatted at 2026-06-20 18:40:30 on dist-test-slave-5lgp"
server_key: "c329ddad5ff3e1114b7510aff6a6dd86"
server_key_iv: "ab3ad24c0bed022ea6893583427c34d1"
server_key_version: "encryptionkey@0"
I20260620 18:40:30.139582 8411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980527431848-22676-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980527431848-22676-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980527431848-22676-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:40:30.156324 8411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:40:30.160900 8411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:40:30.161206 8411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:40:30.179374 8411 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.62:38937
I20260620 18:40:30.179404 8477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.62:38937 every 8 connection(s)
I20260620 18:40:30.180811 8411 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
I20260620 18:40:30.184458 8478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:40:30.185496 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 8411
I20260620 18:40:30.185698 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980527431848-22676-0/minicluster-data/master-0/wal/instance
I20260620 18:40:30.186133 22676 external_mini_cluster.cc:1428] Setting key e903f78775d9cb3b615f3a85dc8cf7ac
I20260620 18:40:30.198285 8478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdbe0c57133444f6b3379a5cf82d6bfc: Bootstrap starting.
Jun 20 18:40:30 dist-test-slave-5lgp krb5kdc[8398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980829, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM
I20260620 18:40:30.202026 8478 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fdbe0c57133444f6b3379a5cf82d6bfc: Neither blocks nor log segments found. Creating new log.
I20260620 18:40:30.203361 8478 log.cc:824] T 00000000000000000000000000000000 P fdbe0c57133444f6b3379a5cf82d6bfc: Log is configured to *not* fsync() on all Append() calls
I20260620 18:40:30.206540 8478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdbe0c57133444f6b3379a5cf82d6bfc: No bootstrap required, opened a new log
I20260620 18:40:30.210120 8481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:30.187903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56418 (local address 127.22.37.62:38937)
0620 18:40:30.188730 (+ 827us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:30.188746 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:30.188807 (+ 61us) server_negotiation.cc:407] Connection header received
0620 18:40:30.190790 (+ 1983us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:30.190810 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:30.191179 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:30.191487 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:40:30.193231 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:30.194581 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:30.196411 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:30.197047 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:30.197170 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:30.202064 (+ 4894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:40:30.202075 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:40:30.202094 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:40:30.202141 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:40:30.205934 (+ 3793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:40:30.206574 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:40:30.206584 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:40:30.206592 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:40:30.206780 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:40:30.207142 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:40:30.207148 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:40:30.207151 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:40:30.207770 (+ 619us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:40:30.208062 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:30.208360 (+ 298us) server_negotiation.cc:299] Negotiation successful
0620 18:40:30.208792 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":352,"threads_started":1}
I20260620 18:40:30.215287 8478 raft_consensus.cc:348] T 00000000000000000000000000000000 P fdbe0c57133444f6b3379a5cf82d6bfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdbe0c57133444f6b3379a5cf82d6bfc" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 38937 } }
I20260620 18:40:30.215667 8478 raft_consensus.cc:374] T 00000000000000000000000000000000 P fdbe0c57133444f6b3379a5cf82d6bfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:40:30.215778 8478 raft_consensus.cc:729] T 00000000000000000000000000000000 P fdbe0c57133444f6b3379a5cf82d6bfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdbe0c57133444f6b3379a5cf82d6bfc, State: Initialized, Role: FOLLOWER
I20260620 18:40:30.216301 8478 consensus_queue.cc:260] T 00000000000000000000000000000000 P fdbe0c57133444f6b3379a5cf82d6bfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdbe0c57133444f6b3379a5cf82d6bfc" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 38937 } }
I20260620 18:40:30.216588 8478 raft_consensus.cc:388] T 00000000000000000000000000000000 P fdbe0c57133444f6b3379a5cf82d6bfc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:40:30.216723 8478 raft_consensus.cc:482] T 00000000000000000000000000000000 P fdbe0c57133444f6b3379a5cf82d6bfc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:40:30.216924 8478 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fdbe0c57133444f6b3379a5cf82d6bfc [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:40:30.218868 8478 raft_consensus.cc:504] T 00000000000000000000000000000000 P fdbe0c57133444f6b3379a5cf82d6bfc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdbe0c57133444f6b3379a5cf82d6bfc" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 38937 } }
I20260620 18:40:30.219326 8478 leader_election.cc:302] T 00000000000000000000000000000000 P fdbe0c57133444f6b3379a5cf82d6bfc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fdbe0c57133444f6b3379a5cf82d6bfc; no voters:
I20260620 18:40:30.219812 8478 leader_election.cc:288] T 00000000000000000000000000000000 P fdbe0c57133444f6b3379a5cf82d6bfc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:40:30.220080 8483 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fdbe0c57133444f6b3379a5cf82d6bfc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:40:30.220789 8483 raft_consensus.cc:686] T 00000000000000000000000000000000 P fdbe0c57133444f6b3379a5cf82d6bfc [term 1 LEADER]: Becoming Leader. State: Replica: fdbe0c57133444f6b3379a5cf82d6bfc, State: Running, Role: LEADER
I20260620 18:40:30.221278 8483 consensus_queue.cc:237] T 00000000000000000000000000000000 P fdbe0c57133444f6b3379a5cf82d6bfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdbe0c57133444f6b3379a5cf82d6bfc" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 38937 } }
I20260620 18:40:30.221973 8478 sys_catalog.cc:564] T 00000000000000000000000000000000 P fdbe0c57133444f6b3379a5cf82d6bfc [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:40:30.224318 8485 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdbe0c57133444f6b3379a5cf82d6bfc [sys.catalog]: SysCatalogTable state changed. Reason: New leader fdbe0c57133444f6b3379a5cf82d6bfc. Latest consensus state: current_term: 1 leader_uuid: "fdbe0c57133444f6b3379a5cf82d6bfc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdbe0c57133444f6b3379a5cf82d6bfc" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 38937 } } }
I20260620 18:40:30.224854 8485 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdbe0c57133444f6b3379a5cf82d6bfc [sys.catalog]: This master's current role is: LEADER
I20260620 18:40:30.224745 8484 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdbe0c57133444f6b3379a5cf82d6bfc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fdbe0c57133444f6b3379a5cf82d6bfc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdbe0c57133444f6b3379a5cf82d6bfc" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 38937 } } }
I20260620 18:40:30.225459 8484 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdbe0c57133444f6b3379a5cf82d6bfc [sys.catalog]: This master's current role is: LEADER
I20260620 18:40:30.233062 8492 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:40:30.237820 8492 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:40:30.247289 8492 catalog_manager.cc:1347] Generated new cluster ID: c6f0568ae1924518a4aef59edab275ae
I20260620 18:40:30.247393 8492 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:40:30.275638 8492 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:40:30.277526 8492 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:40:30.288954 8492 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fdbe0c57133444f6b3379a5cf82d6bfc: Generated new TSK 0
I20260620 18:40:30.290017 8492 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.37.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.37.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.37.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.1.keytab.
Entry for principal kudu/127.22.37.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.1.keytab.
I20260620 18:40:30.348872 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980527431848-22676-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980527431848-22676-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980527431848-22676-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980527431848-22676-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.37.1:0
--local_ip_for_outbound_sockets=127.22.37.1
--webserver_interface=127.22.37.1
--webserver_port=0
--tserver_master_addrs=127.22.37.62:38937
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.1.keytab
--principal=kudu/127.22.37.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:42013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:40:30.683693 8504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:40:30.683986 8504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:40:30.684082 8504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:40:30.691414 8504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:40:30.691531 8504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:40:30.691648 8504 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.37.1
I20260620 18:40:30.701702 8504 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:42013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980527431848-22676-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980527431848-22676-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.1.keytab
--principal=kudu/127.22.37.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.37.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980527431848-22676-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.37.1
--webserver_port=0
--tserver_master_addrs=127.22.37.62:38937
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.37.1
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980527431848-22676-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:40:30.702939 8504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:40:30.704210 8504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:40:30.717823 8509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:40:30.717744 8510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:40:30.717793 8512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:40:30.718990 8504 server_base.cc:1034] running on GCE node
I20260620 18:40:30.719614 8504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:40:30.720520 8504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:40:30.721746 8504 hybrid_clock.cc:648] HybridClock initialized: now 1781980830721711 us; error 63 us; skew 500 ppm
I20260620 18:40:30.722067 8504 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:40:30 dist-test-slave-5lgp krb5kdc[8398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980830, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:40:30.725211 8504 init.cc:377] Logged in from keytab as kudu/127.22.37.1@KRBTEST.COM (short username kudu)
I20260620 18:40:30.726742 8504 webserver.cc:460] Webserver started at http://127.22.37.1:39159/ using document root <none> and password file <none>
I20260620 18:40:30.727420 8504 fs_manager.cc:359] Metadata directory not provided
I20260620 18:40:30.727569 8504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:40:30.727890 8504 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:40:30.730859 8504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980527431848-22676-0/minicluster-data/ts-0/data/instance:
uuid: "646e2af7741644f8867551ba5173926b"
format_stamp: "Formatted at 2026-06-20 18:40:30 on dist-test-slave-5lgp"
server_key: "5bd29d06ba610da1b79e75525f489c1a"
server_key_iv: "970995f0f5f4f57e05a93a305a379f85"
server_key_version: "encryptionkey@0"
I20260620 18:40:30.731622 8504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980527431848-22676-0/minicluster-data/ts-0/wal/instance:
uuid: "646e2af7741644f8867551ba5173926b"
format_stamp: "Formatted at 2026-06-20 18:40:30 on dist-test-slave-5lgp"
server_key: "5bd29d06ba610da1b79e75525f489c1a"
server_key_iv: "970995f0f5f4f57e05a93a305a379f85"
server_key_version: "encryptionkey@0"
I20260620 18:40:30.736668 8504 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 18:40:30.740113 8519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:40:30.741318 8504 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:40:30.741498 8504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980527431848-22676-0/minicluster-data/ts-0/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980527431848-22676-0/minicluster-data/ts-0/wal
uuid: "646e2af7741644f8867551ba5173926b"
format_stamp: "Formatted at 2026-06-20 18:40:30 on dist-test-slave-5lgp"
server_key: "5bd29d06ba610da1b79e75525f489c1a"
server_key_iv: "970995f0f5f4f57e05a93a305a379f85"
server_key_version: "encryptionkey@0"
I20260620 18:40:30.741698 8504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980527431848-22676-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980527431848-22676-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980527431848-22676-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:40:30.758266 8504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:40:30.762060 8504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:40:30.762357 8504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:40:30.763301 8504 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:40:30.764626 8504 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:40:30.764726 8504 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:40:30.764879 8504 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:40:30.764950 8504 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:40:30.797638 8504 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.1:44709
I20260620 18:40:30.797675 8631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.1:44709 every 8 connection(s)
I20260620 18:40:30.798976 8504 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980527431848-22676-0/minicluster-data/ts-0/data/info.pb
I20260620 18:40:30.803153 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 8504
I20260620 18:40:30.803388 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980527431848-22676-0/minicluster-data/ts-0/wal/instance
I20260620 18:40:30.804061 22676 external_mini_cluster.cc:1428] Setting key 71f8b72c904b278b9db45f787562b630
Jun 20 18:40:30 dist-test-slave-5lgp krb5kdc[8398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980830, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.1@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM
I20260620 18:40:30.825021 8634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:30.801649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.1:46199 (local address 127.22.37.62:38937)
0620 18:40:30.802489 (+ 840us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:30.802494 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:30.803587 (+ 1093us) server_negotiation.cc:407] Connection header received
0620 18:40:30.806153 (+ 2566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:30.806159 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:30.806240 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:30.806331 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:40:30.808663 (+ 2332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:30.809441 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:30.812051 (+ 2610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:30.812660 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:30.812785 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:30.817615 (+ 4830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:40:30.817620 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:40:30.817623 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:40:30.817655 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:40:30.821295 (+ 3640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:40:30.822482 (+ 1187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:40:30.822487 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:40:30.822489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:40:30.822618 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:40:30.823123 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:40:30.823129 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:40:30.823132 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:40:30.823484 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:40:30.823650 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:30.824402 (+ 752us) server_negotiation.cc:299] Negotiation successful
0620 18:40:30.824649 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":720,"thread_start_us":610,"threads_started":1}
I20260620 18:40:30.826244 8635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:30.802377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:38937 (local address 127.22.37.1:46199)
0620 18:40:30.803380 (+ 1003us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:30.803459 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0620 18:40:30.805807 (+ 2348us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:40:30.806649 (+ 842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:40:30.806664 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:40:30.807519 (+ 855us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:40:30.808455 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:30.808488 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:30.809614 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:30.809622 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:30.811879 (+ 2257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:30.811899 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:30.812943 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:30.812954 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:30.813304 (+ 350us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:30.814183 (+ 879us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:40:30.814227 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:40:30.817344 (+ 3117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:40:30.821557 (+ 4213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:40:30.821569 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:40:30.821591 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:40:30.822354 (+ 763us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:40:30.822787 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:40:30.822791 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:40:30.822793 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:40:30.822976 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:40:30.823621 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:40:30.823629 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:40:30.823999 (+ 370us) client_negotiation.cc:769] Sending connection context
0620 18:40:30.824295 (+ 296us) client_negotiation.cc:240] Negotiation successful
0620 18:40:30.824880 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"thread_start_us":382,"threads_started":1}
I20260620 18:40:30.827888 8632 heartbeater.cc:347] Connected to a master server at 127.22.37.62:38937
I20260620 18:40:30.828260 8632 heartbeater.cc:464] Registering TS with master...
I20260620 18:40:30.829242 8632 heartbeater.cc:511] Master 127.22.37.62:38937 requested a full tablet report, sending...
I20260620 18:40:30.831699 8443 ts_manager.cc:194] Registered new tserver with Master: 646e2af7741644f8867551ba5173926b (127.22.37.1:44709)
WARNING: no policy specified for kudu/127.22.37.2@KRBTEST.COM; defaulting to no policy
I20260620 18:40:30.833250 8443 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.37.1@KRBTEST.COM'} at 127.22.37.1:46199
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.37.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.37.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.2.keytab.
Entry for principal kudu/127.22.37.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.2.keytab.
I20260620 18:40:30.862272 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980527431848-22676-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980527431848-22676-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980527431848-22676-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980527431848-22676-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.37.2:0
--local_ip_for_outbound_sockets=127.22.37.2
--webserver_interface=127.22.37.2
--webserver_port=0
--tserver_master_addrs=127.22.37.62:38937
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.2.keytab
--principal=kudu/127.22.37.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:42013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:40:31.189958 8638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:40:31.190202 8638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:40:31.190363 8638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:40:31.197464 8638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:40:31.197571 8638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:40:31.197746 8638 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.37.2
I20260620 18:40:31.207453 8638 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:42013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980527431848-22676-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980527431848-22676-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.2.keytab
--principal=kudu/127.22.37.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.37.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980527431848-22676-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.37.2
--webserver_port=0
--tserver_master_addrs=127.22.37.62:38937
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.37.2
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980527431848-22676-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:40:31.208714 8638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:40:31.210098 8638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:40:31.223516 8643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:40:31.223636 8644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:40:31.223487 8646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:40:31.224126 8638 server_base.cc:1034] running on GCE node
I20260620 18:40:31.224843 8638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:40:31.225919 8638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:40:31.227156 8638 hybrid_clock.cc:648] HybridClock initialized: now 1781980831227108 us; error 55 us; skew 500 ppm
I20260620 18:40:31.227501 8638 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:40:31 dist-test-slave-5lgp krb5kdc[8398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980831, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:40:31.230595 8638 init.cc:377] Logged in from keytab as kudu/127.22.37.2@KRBTEST.COM (short username kudu)
I20260620 18:40:31.231928 8638 webserver.cc:460] Webserver started at http://127.22.37.2:41617/ using document root <none> and password file <none>
I20260620 18:40:31.232560 8638 fs_manager.cc:359] Metadata directory not provided
I20260620 18:40:31.232656 8638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:40:31.232910 8638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:40:31.235953 8638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980527431848-22676-0/minicluster-data/ts-1/data/instance:
uuid: "fda09ac9343941d1aba5c6e6d16272e1"
format_stamp: "Formatted at 2026-06-20 18:40:31 on dist-test-slave-5lgp"
server_key: "6e355591f458b70bc6d2653f96fb1d46"
server_key_iv: "889eda9b465b38a838996fd1f4226c4f"
server_key_version: "encryptionkey@0"
I20260620 18:40:31.236663 8638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980527431848-22676-0/minicluster-data/ts-1/wal/instance:
uuid: "fda09ac9343941d1aba5c6e6d16272e1"
format_stamp: "Formatted at 2026-06-20 18:40:31 on dist-test-slave-5lgp"
server_key: "6e355591f458b70bc6d2653f96fb1d46"
server_key_iv: "889eda9b465b38a838996fd1f4226c4f"
server_key_version: "encryptionkey@0"
I20260620 18:40:31.241847 8638 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260620 18:40:31.245813 8653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:40:31.247284 8638 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:40:31.247426 8638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980527431848-22676-0/minicluster-data/ts-1/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980527431848-22676-0/minicluster-data/ts-1/wal
uuid: "fda09ac9343941d1aba5c6e6d16272e1"
format_stamp: "Formatted at 2026-06-20 18:40:31 on dist-test-slave-5lgp"
server_key: "6e355591f458b70bc6d2653f96fb1d46"
server_key_iv: "889eda9b465b38a838996fd1f4226c4f"
server_key_version: "encryptionkey@0"
I20260620 18:40:31.247615 8638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980527431848-22676-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980527431848-22676-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980527431848-22676-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:40:31.270990 8638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:40:31.275605 8638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:40:31.275856 8638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:40:31.276808 8638 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:40:31.278129 8638 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:40:31.278205 8638 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:40:31.278286 8638 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:40:31.278329 8638 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:40:31.311594 8638 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.2:45967
I20260620 18:40:31.311656 8765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.2:45967 every 8 connection(s)
I20260620 18:40:31.312958 8638 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980527431848-22676-0/minicluster-data/ts-1/data/info.pb
I20260620 18:40:31.315999 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 8638
I20260620 18:40:31.316197 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980527431848-22676-0/minicluster-data/ts-1/wal/instance
I20260620 18:40:31.316800 22676 external_mini_cluster.cc:1428] Setting key 441f7fbbde729d21ecf84f15bcd1376c
Jun 20 18:40:31 dist-test-slave-5lgp krb5kdc[8398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980831, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.2@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM
I20260620 18:40:31.335315 8634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:31.315611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.2:33837 (local address 127.22.37.62:38937)
0620 18:40:31.315792 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:31.315796 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:31.317254 (+ 1458us) server_negotiation.cc:407] Connection header received
0620 18:40:31.319683 (+ 2429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:31.319689 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:31.319778 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:31.319888 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:40:31.321404 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:31.322183 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:31.323790 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:31.324261 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:31.324393 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:31.328574 (+ 4181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:40:31.328579 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:40:31.328583 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:40:31.328621 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:40:31.331416 (+ 2795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:40:31.332462 (+ 1046us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:40:31.332466 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:40:31.332469 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:40:31.332616 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:40:31.333174 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:40:31.333178 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:40:31.333180 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:40:31.333570 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:40:31.333708 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:31.334836 (+ 1128us) server_negotiation.cc:299] Negotiation successful
0620 18:40:31.335057 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 18:40:31.336953 8768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:31.316216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:38937 (local address 127.22.37.2:33837)
0620 18:40:31.317079 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:31.317122 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 18:40:31.319379 (+ 2257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:40:31.320126 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:40:31.320135 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:40:31.320653 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:40:31.321218 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:31.321234 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:31.322333 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:31.322345 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:31.323648 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:31.323660 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:31.324502 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:31.324510 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:31.324743 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:31.325506 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:40:31.325529 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:40:31.328309 (+ 2780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:40:31.331580 (+ 3271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:40:31.331593 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:40:31.331626 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:40:31.332303 (+ 677us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:40:31.332752 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:40:31.332760 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:40:31.332766 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:40:31.333030 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:40:31.333709 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:40:31.333720 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:40:31.334343 (+ 623us) client_negotiation.cc:769] Sending connection context
0620 18:40:31.334722 (+ 379us) client_negotiation.cc:240] Negotiation successful
0620 18:40:31.335353 (+ 631us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":345,"threads_started":1}
I20260620 18:40:31.338685 8766 heartbeater.cc:347] Connected to a master server at 127.22.37.62:38937
I20260620 18:40:31.339059 8766 heartbeater.cc:464] Registering TS with master...
I20260620 18:40:31.340065 8766 heartbeater.cc:511] Master 127.22.37.62:38937 requested a full tablet report, sending...
I20260620 18:40:31.342211 8443 ts_manager.cc:194] Registered new tserver with Master: fda09ac9343941d1aba5c6e6d16272e1 (127.22.37.2:45967)
I20260620 18:40:31.343271 8443 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.37.2@KRBTEST.COM'} at 127.22.37.2:33837
WARNING: no policy specified for kudu/127.22.37.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.37.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.37.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.3.keytab.
Entry for principal kudu/127.22.37.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.3.keytab.
I20260620 18:40:31.376266 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980527431848-22676-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980527431848-22676-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980527431848-22676-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980527431848-22676-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.37.3:0
--local_ip_for_outbound_sockets=127.22.37.3
--webserver_interface=127.22.37.3
--webserver_port=0
--tserver_master_addrs=127.22.37.62:38937
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.3.keytab
--principal=kudu/127.22.37.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:42013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:40:31.703061 8771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:40:31.703367 8771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:40:31.703534 8771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:40:31.711217 8771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:40:31.711318 8771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:40:31.711520 8771 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.37.3
I20260620 18:40:31.721589 8771 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:42013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980527431848-22676-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980527431848-22676-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.3.keytab
--principal=kudu/127.22.37.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.37.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980527431848-22676-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.37.3
--webserver_port=0
--tserver_master_addrs=127.22.37.62:38937
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.37.3
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980527431848-22676-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:40:31.722971 8771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:40:31.724272 8771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:40:31.736869 8777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:40:31.736948 8779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:40:31.736901 8776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:40:31.738226 8771 server_base.cc:1034] running on GCE node
I20260620 18:40:31.738811 8771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:40:31.739679 8771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:40:31.740918 8771 hybrid_clock.cc:648] HybridClock initialized: now 1781980831740878 us; error 47 us; skew 500 ppm
I20260620 18:40:31.741231 8771 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:40:31 dist-test-slave-5lgp krb5kdc[8398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980831, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:40:31.744380 8771 init.cc:377] Logged in from keytab as kudu/127.22.37.3@KRBTEST.COM (short username kudu)
I20260620 18:40:31.746035 8771 webserver.cc:460] Webserver started at http://127.22.37.3:41259/ using document root <none> and password file <none>
I20260620 18:40:31.746784 8771 fs_manager.cc:359] Metadata directory not provided
I20260620 18:40:31.746932 8771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:40:31.747244 8771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:40:31.750084 8771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980527431848-22676-0/minicluster-data/ts-2/data/instance:
uuid: "547fc014a3be4a0bb952e293aeb88e42"
format_stamp: "Formatted at 2026-06-20 18:40:31 on dist-test-slave-5lgp"
server_key: "d1e2fac77d98171674fdb11448d69d07"
server_key_iv: "6403268391079bf8697a2e1a51a978af"
server_key_version: "encryptionkey@0"
I20260620 18:40:31.750942 8771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980527431848-22676-0/minicluster-data/ts-2/wal/instance:
uuid: "547fc014a3be4a0bb952e293aeb88e42"
format_stamp: "Formatted at 2026-06-20 18:40:31 on dist-test-slave-5lgp"
server_key: "d1e2fac77d98171674fdb11448d69d07"
server_key_iv: "6403268391079bf8697a2e1a51a978af"
server_key_version: "encryptionkey@0"
I20260620 18:40:31.756072 8771 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 18:40:31.759361 8786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:40:31.760566 8771 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 18:40:31.760758 8771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980527431848-22676-0/minicluster-data/ts-2/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980527431848-22676-0/minicluster-data/ts-2/wal
uuid: "547fc014a3be4a0bb952e293aeb88e42"
format_stamp: "Formatted at 2026-06-20 18:40:31 on dist-test-slave-5lgp"
server_key: "d1e2fac77d98171674fdb11448d69d07"
server_key_iv: "6403268391079bf8697a2e1a51a978af"
server_key_version: "encryptionkey@0"
I20260620 18:40:31.760955 8771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980527431848-22676-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980527431848-22676-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980527431848-22676-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:40:31.779183 8771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:40:31.783106 8771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:40:31.783392 8771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:40:31.784332 8771 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:40:31.785715 8771 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:40:31.785821 8771 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:40:31.785976 8771 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:40:31.786053 8771 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:40:31.819895 8771 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.3:36357
I20260620 18:40:31.819994 8898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.3:36357 every 8 connection(s)
I20260620 18:40:31.821276 8771 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980527431848-22676-0/minicluster-data/ts-2/data/info.pb
I20260620 18:40:31.830389 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 8771
I20260620 18:40:31.830569 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980527431848-22676-0/minicluster-data/ts-2/wal/instance
I20260620 18:40:31.831127 22676 external_mini_cluster.cc:1428] Setting key fbc8d0ed57b23d3c5ed79b3e62fcb72d
I20260620 18:40:31.836673 8632 heartbeater.cc:503] Master 127.22.37.62:38937 was elected leader, sending a full tablet report...
Jun 20 18:40:31 dist-test-slave-5lgp krb5kdc[8398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980831, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.3@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM
I20260620 18:40:31.845369 8634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:31.825601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.3:54245 (local address 127.22.37.62:38937)
0620 18:40:31.825842 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:31.825848 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:31.826491 (+ 643us) server_negotiation.cc:407] Connection header received
0620 18:40:31.829736 (+ 3245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:31.829744 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:31.829840 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:31.829951 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:40:31.833317 (+ 3366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:31.834282 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:31.835724 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:31.836113 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:31.836205 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:31.839786 (+ 3581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:40:31.839791 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:40:31.839795 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:40:31.839825 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:40:31.842307 (+ 2482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:40:31.842952 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:40:31.842955 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:40:31.842958 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:40:31.843101 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:40:31.843513 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:40:31.843516 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:40:31.843519 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:40:31.843792 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:40:31.843973 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:31.844762 (+ 789us) server_negotiation.cc:299] Negotiation successful
0620 18:40:31.845007 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260620 18:40:31.846647 8901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:31.825201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:38937 (local address 127.22.37.3:54245)
0620 18:40:31.826262 (+ 1061us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:31.826343 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0620 18:40:31.829313 (+ 2970us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:40:31.831823 (+ 2510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:40:31.831833 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:40:31.832446 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:40:31.833162 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:31.833182 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:31.834393 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:31.834397 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:31.835603 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:31.835614 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:31.836291 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:31.836295 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:31.836479 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:31.837197 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:40:31.837219 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:40:31.839601 (+ 2382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:40:31.842436 (+ 2835us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:40:31.842443 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:40:31.842455 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:40:31.842852 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:40:31.843242 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:40:31.843246 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:40:31.843248 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:40:31.843421 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:40:31.843963 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:40:31.843968 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:40:31.844403 (+ 435us) client_negotiation.cc:769] Sending connection context
0620 18:40:31.844676 (+ 273us) client_negotiation.cc:240] Negotiation successful
0620 18:40:31.845185 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":691,"thread_start_us":403,"threads_started":1}
I20260620 18:40:31.847983 8899 heartbeater.cc:347] Connected to a master server at 127.22.37.62:38937
I20260620 18:40:31.848281 8899 heartbeater.cc:464] Registering TS with master...
I20260620 18:40:31.848969 8899 heartbeater.cc:511] Master 127.22.37.62:38937 requested a full tablet report, sending...
I20260620 18:40:31.850399 8443 ts_manager.cc:194] Registered new tserver with Master: 547fc014a3be4a0bb952e293aeb88e42 (127.22.37.3:36357)
I20260620 18:40:31.851276 8443 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.37.3@KRBTEST.COM'} at 127.22.37.3:54245
I20260620 18:40:31.858834 22676 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:40:31.877239 8634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:31.863173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56422 (local address 127.22.37.62:38937)
0620 18:40:31.863341 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:31.863346 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:31.863863 (+ 517us) server_negotiation.cc:407] Connection header received
0620 18:40:31.864076 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:31.864080 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:31.864158 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:31.864263 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:40:31.865675 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:31.866385 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:31.867758 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:31.868183 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:31.868313 (+ 130us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:31.871035 (+ 2722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:40:31.871045 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:40:31.871049 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:40:31.871085 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:40:31.874079 (+ 2994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:40:31.875026 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:40:31.875031 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:40:31.875033 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:40:31.875229 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:40:31.875781 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:40:31.875786 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:40:31.875790 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:40:31.876105 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:40:31.876380 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:31.876769 (+ 389us) server_negotiation.cc:299] Negotiation successful
0620 18:40:31.876944 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 18:40:31.881019 8443 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56422:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:40:31.887820 8443 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:40:31.910030 8911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:31.899607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.2:45967 (local address 127.0.0.1:45348)
0620 18:40:31.900640 (+ 1033us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:31.900710 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0620 18:40:31.900924 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:40:31.902226 (+ 1302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:40:31.902234 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:40:31.902263 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:40:31.902748 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:31.902768 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:31.905086 (+ 2318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:31.905094 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:31.907470 (+ 2376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:31.907482 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:31.908826 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:31.908833 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:31.909190 (+ 357us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:31.909262 (+ 72us) client_negotiation.cc:769] Sending connection context
0620 18:40:31.909437 (+ 175us) client_negotiation.cc:240] Negotiation successful
0620 18:40:31.909630 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":697,"thread_start_us":540,"threads_started":1}
I20260620 18:40:31.910472 8913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:31.899793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45348 (local address 127.22.37.2:45967)
0620 18:40:31.901531 (+ 1738us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:31.901542 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:31.901587 (+ 45us) server_negotiation.cc:407] Connection header received
0620 18:40:31.901692 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:31.901701 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:31.901975 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:31.903142 (+ 1167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:40:31.903377 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:31.904952 (+ 1575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:31.907625 (+ 2673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:31.908621 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:31.908754 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:31.908955 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:31.909976 (+ 1021us) server_negotiation.cc:299] Negotiation successful
0620 18:40:31.910149 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1545,"thread_start_us":1444,"threads_started":1}
I20260620 18:40:31.912128 8912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:31.899760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.3:36357 (local address 127.0.0.1:38636)
0620 18:40:31.902297 (+ 2537us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:31.902318 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 18:40:31.902444 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:40:31.903379 (+ 935us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:40:31.903388 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:40:31.903413 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:40:31.903919 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:31.903931 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:31.905912 (+ 1981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:31.905916 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:31.908343 (+ 2427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:31.908358 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:31.910426 (+ 2068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:31.910433 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:31.911458 (+ 1025us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:31.911490 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 18:40:31.911589 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 18:40:31.911721 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2418,"mutex_wait_us":26,"thread_start_us":2266,"threads_started":1}
I20260620 18:40:31.912209 8916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:31.902217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32836 (local address 127.22.37.1:44709)
0620 18:40:31.903314 (+ 1097us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:31.903327 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:31.903919 (+ 592us) server_negotiation.cc:407] Connection header received
0620 18:40:31.904179 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:31.904187 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:31.904410 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:31.904588 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:40:31.905333 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:31.907021 (+ 1688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:31.909944 (+ 2923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:31.911168 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:31.911318 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:31.911534 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:31.911729 (+ 195us) server_negotiation.cc:299] Negotiation successful
0620 18:40:31.911888 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":945,"thread_start_us":798,"threads_started":1}
I20260620 18:40:31.912535 8915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:31.901610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.1:44709 (local address 127.0.0.1:32836)
0620 18:40:31.903802 (+ 2192us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:31.903816 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 18:40:31.903967 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:40:31.904611 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:40:31.904616 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:40:31.904641 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:40:31.904975 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:31.904985 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:31.907171 (+ 2186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:31.907178 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:31.909766 (+ 2588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:31.909786 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:31.911363 (+ 1577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:31.911369 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:31.911533 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:31.911572 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 18:40:31.911690 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 18:40:31.911814 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2063,"mutex_wait_us":1,"thread_start_us":1925,"threads_started":1}
I20260620 18:40:31.912881 8914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:31.900206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38636 (local address 127.22.37.3:36357)
0620 18:40:31.901885 (+ 1679us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:31.901893 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:31.902564 (+ 671us) server_negotiation.cc:407] Connection header received
0620 18:40:31.902670 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:31.902680 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:31.902923 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:31.903115 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:40:31.904251 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:31.905759 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:31.908511 (+ 2752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:31.909928 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:31.910048 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:31.910846 (+ 798us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:31.912418 (+ 1572us) server_negotiation.cc:299] Negotiation successful
0620 18:40:31.912576 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1532,"thread_start_us":1388,"threads_started":1}
I20260620 18:40:31.914564 8701 tablet_service.cc:1467] Processing CreateTablet for tablet 36e93b81911e4fd0be232dd0ca8cf9cb (DEFAULT_TABLE table=test-table [id=5009dbc1c2e7481daf78e77582027392]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:40:31.916062 8701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36e93b81911e4fd0be232dd0ca8cf9cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:40:31.915817 8567 tablet_service.cc:1467] Processing CreateTablet for tablet 36e93b81911e4fd0be232dd0ca8cf9cb (DEFAULT_TABLE table=test-table [id=5009dbc1c2e7481daf78e77582027392]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:40:31.917069 8567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36e93b81911e4fd0be232dd0ca8cf9cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:40:31.916635 8834 tablet_service.cc:1467] Processing CreateTablet for tablet 36e93b81911e4fd0be232dd0ca8cf9cb (DEFAULT_TABLE table=test-table [id=5009dbc1c2e7481daf78e77582027392]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:40:31.918022 8834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36e93b81911e4fd0be232dd0ca8cf9cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:40:31.928668 8917 tablet_bootstrap.cc:492] T 36e93b81911e4fd0be232dd0ca8cf9cb P 646e2af7741644f8867551ba5173926b: Bootstrap starting.
I20260620 18:40:31.932459 8917 tablet_bootstrap.cc:654] T 36e93b81911e4fd0be232dd0ca8cf9cb P 646e2af7741644f8867551ba5173926b: Neither blocks nor log segments found. Creating new log.
I20260620 18:40:31.933753 8917 log.cc:824] T 36e93b81911e4fd0be232dd0ca8cf9cb P 646e2af7741644f8867551ba5173926b: Log is configured to *not* fsync() on all Append() calls
I20260620 18:40:31.934954 8919 tablet_bootstrap.cc:492] T 36e93b81911e4fd0be232dd0ca8cf9cb P 547fc014a3be4a0bb952e293aeb88e42: Bootstrap starting.
I20260620 18:40:31.936198 8918 tablet_bootstrap.cc:492] T 36e93b81911e4fd0be232dd0ca8cf9cb P fda09ac9343941d1aba5c6e6d16272e1: Bootstrap starting.
I20260620 18:40:31.937000 8917 tablet_bootstrap.cc:492] T 36e93b81911e4fd0be232dd0ca8cf9cb P 646e2af7741644f8867551ba5173926b: No bootstrap required, opened a new log
I20260620 18:40:31.937279 8917 ts_tablet_manager.cc:1397] T 36e93b81911e4fd0be232dd0ca8cf9cb P 646e2af7741644f8867551ba5173926b: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260620 18:40:31.938678 8919 tablet_bootstrap.cc:654] T 36e93b81911e4fd0be232dd0ca8cf9cb P 547fc014a3be4a0bb952e293aeb88e42: Neither blocks nor log segments found. Creating new log.
I20260620 18:40:31.939647 8918 tablet_bootstrap.cc:654] T 36e93b81911e4fd0be232dd0ca8cf9cb P fda09ac9343941d1aba5c6e6d16272e1: Neither blocks nor log segments found. Creating new log.
I20260620 18:40:31.939949 8919 log.cc:824] T 36e93b81911e4fd0be232dd0ca8cf9cb P 547fc014a3be4a0bb952e293aeb88e42: Log is configured to *not* fsync() on all Append() calls
I20260620 18:40:31.940933 8918 log.cc:824] T 36e93b81911e4fd0be232dd0ca8cf9cb P fda09ac9343941d1aba5c6e6d16272e1: Log is configured to *not* fsync() on all Append() calls
I20260620 18:40:31.944548 8919 tablet_bootstrap.cc:492] T 36e93b81911e4fd0be232dd0ca8cf9cb P 547fc014a3be4a0bb952e293aeb88e42: No bootstrap required, opened a new log
I20260620 18:40:31.944733 8918 tablet_bootstrap.cc:492] T 36e93b81911e4fd0be232dd0ca8cf9cb P fda09ac9343941d1aba5c6e6d16272e1: No bootstrap required, opened a new log
I20260620 18:40:31.944844 8919 ts_tablet_manager.cc:1397] T 36e93b81911e4fd0be232dd0ca8cf9cb P 547fc014a3be4a0bb952e293aeb88e42: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260620 18:40:31.944988 8918 ts_tablet_manager.cc:1397] T 36e93b81911e4fd0be232dd0ca8cf9cb P fda09ac9343941d1aba5c6e6d16272e1: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260620 18:40:31.946643 8917 raft_consensus.cc:348] T 36e93b81911e4fd0be232dd0ca8cf9cb P 646e2af7741644f8867551ba5173926b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fda09ac9343941d1aba5c6e6d16272e1" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 45967 } } peers { permanent_uuid: "547fc014a3be4a0bb952e293aeb88e42" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 36357 } } peers { permanent_uuid: "646e2af7741644f8867551ba5173926b" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44709 } }
I20260620 18:40:31.947098 8917 raft_consensus.cc:374] T 36e93b81911e4fd0be232dd0ca8cf9cb P 646e2af7741644f8867551ba5173926b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:40:31.947196 8917 raft_consensus.cc:729] T 36e93b81911e4fd0be232dd0ca8cf9cb P 646e2af7741644f8867551ba5173926b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 646e2af7741644f8867551ba5173926b, State: Initialized, Role: FOLLOWER
I20260620 18:40:31.947825 8917 consensus_queue.cc:260] T 36e93b81911e4fd0be232dd0ca8cf9cb P 646e2af7741644f8867551ba5173926b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fda09ac9343941d1aba5c6e6d16272e1" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 45967 } } peers { permanent_uuid: "547fc014a3be4a0bb952e293aeb88e42" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 36357 } } peers { permanent_uuid: "646e2af7741644f8867551ba5173926b" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44709 } }
I20260620 18:40:31.949651 8917 ts_tablet_manager.cc:1428] T 36e93b81911e4fd0be232dd0ca8cf9cb P 646e2af7741644f8867551ba5173926b: Time spent starting tablet: real 0.012s user 0.009s sys 0.004s
I20260620 18:40:31.954433 8919 raft_consensus.cc:348] T 36e93b81911e4fd0be232dd0ca8cf9cb P 547fc014a3be4a0bb952e293aeb88e42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fda09ac9343941d1aba5c6e6d16272e1" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 45967 } } peers { permanent_uuid: "547fc014a3be4a0bb952e293aeb88e42" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 36357 } } peers { permanent_uuid: "646e2af7741644f8867551ba5173926b" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44709 } }
I20260620 18:40:31.954809 8919 raft_consensus.cc:374] T 36e93b81911e4fd0be232dd0ca8cf9cb P 547fc014a3be4a0bb952e293aeb88e42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:40:31.954909 8919 raft_consensus.cc:729] T 36e93b81911e4fd0be232dd0ca8cf9cb P 547fc014a3be4a0bb952e293aeb88e42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 547fc014a3be4a0bb952e293aeb88e42, State: Initialized, Role: FOLLOWER
I20260620 18:40:31.955367 8918 raft_consensus.cc:348] T 36e93b81911e4fd0be232dd0ca8cf9cb P fda09ac9343941d1aba5c6e6d16272e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fda09ac9343941d1aba5c6e6d16272e1" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 45967 } } peers { permanent_uuid: "547fc014a3be4a0bb952e293aeb88e42" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 36357 } } peers { permanent_uuid: "646e2af7741644f8867551ba5173926b" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44709 } }
I20260620 18:40:31.955708 8918 raft_consensus.cc:374] T 36e93b81911e4fd0be232dd0ca8cf9cb P fda09ac9343941d1aba5c6e6d16272e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:40:31.955575 8919 consensus_queue.cc:260] T 36e93b81911e4fd0be232dd0ca8cf9cb P 547fc014a3be4a0bb952e293aeb88e42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fda09ac9343941d1aba5c6e6d16272e1" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 45967 } } peers { permanent_uuid: "547fc014a3be4a0bb952e293aeb88e42" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 36357 } } peers { permanent_uuid: "646e2af7741644f8867551ba5173926b" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44709 } }
I20260620 18:40:31.955801 8918 raft_consensus.cc:729] T 36e93b81911e4fd0be232dd0ca8cf9cb P fda09ac9343941d1aba5c6e6d16272e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fda09ac9343941d1aba5c6e6d16272e1, State: Initialized, Role: FOLLOWER
I20260620 18:40:31.956336 8918 consensus_queue.cc:260] T 36e93b81911e4fd0be232dd0ca8cf9cb P fda09ac9343941d1aba5c6e6d16272e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fda09ac9343941d1aba5c6e6d16272e1" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 45967 } } peers { permanent_uuid: "547fc014a3be4a0bb952e293aeb88e42" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 36357 } } peers { permanent_uuid: "646e2af7741644f8867551ba5173926b" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44709 } }
I20260620 18:40:31.956795 8899 heartbeater.cc:503] Master 127.22.37.62:38937 was elected leader, sending a full tablet report...
I20260620 18:40:31.957396 8766 heartbeater.cc:503] Master 127.22.37.62:38937 was elected leader, sending a full tablet report...
I20260620 18:40:31.957450 8919 ts_tablet_manager.cc:1428] T 36e93b81911e4fd0be232dd0ca8cf9cb P 547fc014a3be4a0bb952e293aeb88e42: Time spent starting tablet: real 0.012s user 0.009s sys 0.006s
I20260620 18:40:31.958020 8918 ts_tablet_manager.cc:1428] T 36e93b81911e4fd0be232dd0ca8cf9cb P fda09ac9343941d1aba5c6e6d16272e1: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260620 18:40:31.973345 8924 raft_consensus.cc:482] T 36e93b81911e4fd0be232dd0ca8cf9cb P 547fc014a3be4a0bb952e293aeb88e42 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:40:31.973695 8924 raft_consensus.cc:504] T 36e93b81911e4fd0be232dd0ca8cf9cb P 547fc014a3be4a0bb952e293aeb88e42 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fda09ac9343941d1aba5c6e6d16272e1" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 45967 } } peers { permanent_uuid: "547fc014a3be4a0bb952e293aeb88e42" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 36357 } } peers { permanent_uuid: "646e2af7741644f8867551ba5173926b" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44709 } }
I20260620 18:40:31.975282 8924 leader_election.cc:288] T 36e93b81911e4fd0be232dd0ca8cf9cb P 547fc014a3be4a0bb952e293aeb88e42 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fda09ac9343941d1aba5c6e6d16272e1 (127.22.37.2:45967), 646e2af7741644f8867551ba5173926b (127.22.37.1:44709)
I20260620 18:40:31.981053 8913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:31.975785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.3:42349 (local address 127.22.37.2:45967)
0620 18:40:31.975996 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:31.976003 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:31.976110 (+ 107us) server_negotiation.cc:407] Connection header received
0620 18:40:31.976375 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:31.976384 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:31.976480 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:31.976595 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:40:31.977071 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:31.977856 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:31.979430 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:31.980186 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:31.980324 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:31.980415 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:31.980683 (+ 268us) server_negotiation.cc:299] Negotiation successful
0620 18:40:31.980803 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 18:40:31.981053 8901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:31.975771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.2:45967 (local address 127.22.37.3:42349)
0620 18:40:31.975994 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:31.976013 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:40:31.976205 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:40:31.976763 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:40:31.976767 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:40:31.976787 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:40:31.976938 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:31.976944 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:31.977980 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:31.977988 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:31.979267 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:31.979278 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:31.980343 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:31.980349 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:31.980517 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:31.980540 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 18:40:31.980646 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 18:40:31.980785 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260620 18:40:31.982205 8926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:31.976155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.1:44709 (local address 127.22.37.3:33381)
0620 18:40:31.976813 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:31.976828 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 18:40:31.976967 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:40:31.977278 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:40:31.977283 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:40:31.977305 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:40:31.977645 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:31.977653 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:31.978634 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:31.978639 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:31.979950 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:31.979961 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:31.981501 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:31.981507 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:31.981688 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:31.981719 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 18:40:31.981815 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 18:40:31.981947 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":414,"threads_started":1}
I20260620 18:40:31.982273 8916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:31.976329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.3:33381 (local address 127.22.37.1:44709)
0620 18:40:31.976521 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:31.976528 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:31.976914 (+ 386us) server_negotiation.cc:407] Connection header received
0620 18:40:31.977108 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:31.977117 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:31.977172 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:31.977267 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:40:31.977798 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:31.978484 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:31.980110 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:31.981143 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:31.981269 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:31.981357 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:31.981862 (+ 505us) server_negotiation.cc:299] Negotiation successful
0620 18:40:31.982034 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 18:40:31.982218 8721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36e93b81911e4fd0be232dd0ca8cf9cb" candidate_uuid: "547fc014a3be4a0bb952e293aeb88e42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fda09ac9343941d1aba5c6e6d16272e1" is_pre_election: true
I20260620 18:40:31.982815 8721 raft_consensus.cc:2454] T 36e93b81911e4fd0be232dd0ca8cf9cb P fda09ac9343941d1aba5c6e6d16272e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 547fc014a3be4a0bb952e293aeb88e42 in term 0.
I20260620 18:40:31.983232 8587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36e93b81911e4fd0be232dd0ca8cf9cb" candidate_uuid: "547fc014a3be4a0bb952e293aeb88e42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "646e2af7741644f8867551ba5173926b" is_pre_election: true
I20260620 18:40:31.983462 8787 leader_election.cc:302] T 36e93b81911e4fd0be232dd0ca8cf9cb P 547fc014a3be4a0bb952e293aeb88e42 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 547fc014a3be4a0bb952e293aeb88e42, fda09ac9343941d1aba5c6e6d16272e1; no voters:
I20260620 18:40:31.983757 8587 raft_consensus.cc:2454] T 36e93b81911e4fd0be232dd0ca8cf9cb P 646e2af7741644f8867551ba5173926b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 547fc014a3be4a0bb952e293aeb88e42 in term 0.
I20260620 18:40:31.983892 8924 raft_consensus.cc:2781] T 36e93b81911e4fd0be232dd0ca8cf9cb P 547fc014a3be4a0bb952e293aeb88e42 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:40:31.984023 8924 raft_consensus.cc:482] T 36e93b81911e4fd0be232dd0ca8cf9cb P 547fc014a3be4a0bb952e293aeb88e42 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:40:31.984138 8924 raft_consensus.cc:3037] T 36e93b81911e4fd0be232dd0ca8cf9cb P 547fc014a3be4a0bb952e293aeb88e42 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:40:31.986403 8924 raft_consensus.cc:504] T 36e93b81911e4fd0be232dd0ca8cf9cb P 547fc014a3be4a0bb952e293aeb88e42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fda09ac9343941d1aba5c6e6d16272e1" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 45967 } } peers { permanent_uuid: "547fc014a3be4a0bb952e293aeb88e42" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 36357 } } peers { permanent_uuid: "646e2af7741644f8867551ba5173926b" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44709 } }
I20260620 18:40:31.986965 8924 leader_election.cc:288] T 36e93b81911e4fd0be232dd0ca8cf9cb P 547fc014a3be4a0bb952e293aeb88e42 [CANDIDATE]: Term 1 election: Requested vote from peers fda09ac9343941d1aba5c6e6d16272e1 (127.22.37.2:45967), 646e2af7741644f8867551ba5173926b (127.22.37.1:44709)
I20260620 18:40:31.987392 8721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36e93b81911e4fd0be232dd0ca8cf9cb" candidate_uuid: "547fc014a3be4a0bb952e293aeb88e42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fda09ac9343941d1aba5c6e6d16272e1"
I20260620 18:40:31.987468 8587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36e93b81911e4fd0be232dd0ca8cf9cb" candidate_uuid: "547fc014a3be4a0bb952e293aeb88e42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "646e2af7741644f8867551ba5173926b"
I20260620 18:40:31.987577 8721 raft_consensus.cc:3037] T 36e93b81911e4fd0be232dd0ca8cf9cb P fda09ac9343941d1aba5c6e6d16272e1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:40:31.987706 8587 raft_consensus.cc:3037] T 36e93b81911e4fd0be232dd0ca8cf9cb P 646e2af7741644f8867551ba5173926b [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:40:31.989898 8721 raft_consensus.cc:2454] T 36e93b81911e4fd0be232dd0ca8cf9cb P fda09ac9343941d1aba5c6e6d16272e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 547fc014a3be4a0bb952e293aeb88e42 in term 1.
I20260620 18:40:31.989904 8587 raft_consensus.cc:2454] T 36e93b81911e4fd0be232dd0ca8cf9cb P 646e2af7741644f8867551ba5173926b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 547fc014a3be4a0bb952e293aeb88e42 in term 1.
I20260620 18:40:31.990289 8787 leader_election.cc:302] T 36e93b81911e4fd0be232dd0ca8cf9cb P 547fc014a3be4a0bb952e293aeb88e42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 547fc014a3be4a0bb952e293aeb88e42, fda09ac9343941d1aba5c6e6d16272e1; no voters:
I20260620 18:40:31.990579 8924 raft_consensus.cc:2781] T 36e93b81911e4fd0be232dd0ca8cf9cb P 547fc014a3be4a0bb952e293aeb88e42 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:40:31.991014 8924 raft_consensus.cc:686] T 36e93b81911e4fd0be232dd0ca8cf9cb P 547fc014a3be4a0bb952e293aeb88e42 [term 1 LEADER]: Becoming Leader. State: Replica: 547fc014a3be4a0bb952e293aeb88e42, State: Running, Role: LEADER
I20260620 18:40:31.991567 8924 consensus_queue.cc:237] T 36e93b81911e4fd0be232dd0ca8cf9cb P 547fc014a3be4a0bb952e293aeb88e42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fda09ac9343941d1aba5c6e6d16272e1" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 45967 } } peers { permanent_uuid: "547fc014a3be4a0bb952e293aeb88e42" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 36357 } } peers { permanent_uuid: "646e2af7741644f8867551ba5173926b" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44709 } }
I20260620 18:40:31.996001 8443 catalog_manager.cc:5495] T 36e93b81911e4fd0be232dd0ca8cf9cb P 547fc014a3be4a0bb952e293aeb88e42 reported cstate change: term changed from 0 to 1, leader changed from <none> to 547fc014a3be4a0bb952e293aeb88e42 (127.22.37.3). New cstate: current_term: 1 leader_uuid: "547fc014a3be4a0bb952e293aeb88e42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fda09ac9343941d1aba5c6e6d16272e1" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 45967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "547fc014a3be4a0bb952e293aeb88e42" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 36357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "646e2af7741644f8867551ba5173926b" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44709 } health_report { overall_health: UNKNOWN } } }
I20260620 18:40:32.012089 8914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:32.007187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38640 (local address 127.22.37.3:36357)
0620 18:40:32.007376 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:32.007381 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:32.007419 (+ 38us) server_negotiation.cc:407] Connection header received
0620 18:40:32.007493 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:32.007496 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:32.007570 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:32.007690 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:40:32.008001 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:32.008716 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:32.010093 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:32.010466 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:32.010562 (+ 96us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:32.010986 (+ 424us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:40:32.011475 (+ 489us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:40:32.011605 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:32.011734 (+ 129us) server_negotiation.cc:299] Negotiation successful
0620 18:40:32.011800 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 18:40:32.020102 8587 raft_consensus.cc:1261] T 36e93b81911e4fd0be232dd0ca8cf9cb P 646e2af7741644f8867551ba5173926b [term 1 FOLLOWER]: Refusing update from remote peer 547fc014a3be4a0bb952e293aeb88e42: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:40:32.020102 8721 raft_consensus.cc:1261] T 36e93b81911e4fd0be232dd0ca8cf9cb P fda09ac9343941d1aba5c6e6d16272e1 [term 1 FOLLOWER]: Refusing update from remote peer 547fc014a3be4a0bb952e293aeb88e42: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:40:32.021090 8927 consensus_queue.cc:1035] T 36e93b81911e4fd0be232dd0ca8cf9cb P 547fc014a3be4a0bb952e293aeb88e42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "646e2af7741644f8867551ba5173926b" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 44709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:40:32.021495 8924 consensus_queue.cc:1035] T 36e93b81911e4fd0be232dd0ca8cf9cb P 547fc014a3be4a0bb952e293aeb88e42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fda09ac9343941d1aba5c6e6d16272e1" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 45967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:40:32.048817 8443 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56422:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:40:32.049098 8443 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56422:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:40:32.052983 8443 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fdbe0c57133444f6b3379a5cf82d6bfc: Sending DeleteTablet for 3 replicas of tablet 36e93b81911e4fd0be232dd0ca8cf9cb
I20260620 18:40:32.054085 8701 tablet_service.cc:1514] Processing DeleteTablet for tablet 36e93b81911e4fd0be232dd0ca8cf9cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:40:32 UTC) from {username='kudu', principal='kudu/127.22.37.62@KRBTEST.COM'} at 127.0.0.1:45348
I20260620 18:40:32.054118 8834 tablet_service.cc:1514] Processing DeleteTablet for tablet 36e93b81911e4fd0be232dd0ca8cf9cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:40:32 UTC) from {username='kudu', principal='kudu/127.22.37.62@KRBTEST.COM'} at 127.0.0.1:38636
I20260620 18:40:32.054167 8567 tablet_service.cc:1514] Processing DeleteTablet for tablet 36e93b81911e4fd0be232dd0ca8cf9cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:40:32 UTC) from {username='kudu', principal='kudu/127.22.37.62@KRBTEST.COM'} at 127.0.0.1:32836
I20260620 18:40:32.059648 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 8504
I20260620 18:40:32.065928 8941 tablet_replica.cc:330] T 36e93b81911e4fd0be232dd0ca8cf9cb P 646e2af7741644f8867551ba5173926b: stopping tablet replica
I20260620 18:40:32.066429 8941 raft_consensus.cc:2229] T 36e93b81911e4fd0be232dd0ca8cf9cb P 646e2af7741644f8867551ba5173926b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:40:32.067024 8941 raft_consensus.cc:2258] T 36e93b81911e4fd0be232dd0ca8cf9cb P 646e2af7741644f8867551ba5173926b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:40:32.067824 8942 tablet_replica.cc:330] T 36e93b81911e4fd0be232dd0ca8cf9cb P 547fc014a3be4a0bb952e293aeb88e42: stopping tablet replica
I20260620 18:40:32.068137 8942 raft_consensus.cc:2229] T 36e93b81911e4fd0be232dd0ca8cf9cb P 547fc014a3be4a0bb952e293aeb88e42 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:40:32.068629 8942 raft_consensus.cc:2258] T 36e93b81911e4fd0be232dd0ca8cf9cb P 547fc014a3be4a0bb952e293aeb88e42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:40:32.070425 8941 ts_tablet_manager.cc:1905] T 36e93b81911e4fd0be232dd0ca8cf9cb P 646e2af7741644f8867551ba5173926b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:40:32.074520 8942 ts_tablet_manager.cc:1905] T 36e93b81911e4fd0be232dd0ca8cf9cb P 547fc014a3be4a0bb952e293aeb88e42: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:40:32.075964 8941 ts_tablet_manager.cc:1918] T 36e93b81911e4fd0be232dd0ca8cf9cb P 646e2af7741644f8867551ba5173926b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:40:32.076119 8941 log.cc:1196] T 36e93b81911e4fd0be232dd0ca8cf9cb P 646e2af7741644f8867551ba5173926b: Deleting WAL directory at /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980527431848-22676-0/minicluster-data/ts-0/wal/wals/36e93b81911e4fd0be232dd0ca8cf9cb
I20260620 18:40:32.076691 8940 tablet_replica.cc:330] T 36e93b81911e4fd0be232dd0ca8cf9cb P fda09ac9343941d1aba5c6e6d16272e1: stopping tablet replica
I20260620 18:40:32.076748 8941 ts_tablet_manager.cc:1939] T 36e93b81911e4fd0be232dd0ca8cf9cb P 646e2af7741644f8867551ba5173926b: Deleting consensus metadata
I20260620 18:40:32.077206 8940 raft_consensus.cc:2229] T 36e93b81911e4fd0be232dd0ca8cf9cb P fda09ac9343941d1aba5c6e6d16272e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:40:32.077733 8940 raft_consensus.cc:2258] T 36e93b81911e4fd0be232dd0ca8cf9cb P fda09ac9343941d1aba5c6e6d16272e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:40:32.078519 8427 catalog_manager.cc:4841] TS 646e2af7741644f8867551ba5173926b (127.22.37.1:44709): tablet 36e93b81911e4fd0be232dd0ca8cf9cb (table test-table [id=5009dbc1c2e7481daf78e77582027392]) successfully deleted
I20260620 18:40:32.082901 8942 ts_tablet_manager.cc:1918] T 36e93b81911e4fd0be232dd0ca8cf9cb P 547fc014a3be4a0bb952e293aeb88e42: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:40:32.083138 8942 log.cc:1196] T 36e93b81911e4fd0be232dd0ca8cf9cb P 547fc014a3be4a0bb952e293aeb88e42: Deleting WAL directory at /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980527431848-22676-0/minicluster-data/ts-2/wal/wals/36e93b81911e4fd0be232dd0ca8cf9cb
I20260620 18:40:32.083176 8940 ts_tablet_manager.cc:1905] T 36e93b81911e4fd0be232dd0ca8cf9cb P fda09ac9343941d1aba5c6e6d16272e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:40:32.083791 8942 ts_tablet_manager.cc:1939] T 36e93b81911e4fd0be232dd0ca8cf9cb P 547fc014a3be4a0bb952e293aeb88e42: Deleting consensus metadata
I20260620 18:40:32.085520 8430 catalog_manager.cc:4841] TS 547fc014a3be4a0bb952e293aeb88e42 (127.22.37.3:36357): tablet 36e93b81911e4fd0be232dd0ca8cf9cb (table test-table [id=5009dbc1c2e7481daf78e77582027392]) successfully deleted
Jun 20 18:40:32 dist-test-slave-5lgp krb5kdc[8398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980829, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.37.1@KRBTEST.COM
I20260620 18:40:32.089278 8940 ts_tablet_manager.cc:1918] T 36e93b81911e4fd0be232dd0ca8cf9cb P fda09ac9343941d1aba5c6e6d16272e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:40:32.089550 8940 log.cc:1196] T 36e93b81911e4fd0be232dd0ca8cf9cb P fda09ac9343941d1aba5c6e6d16272e1: Deleting WAL directory at /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980527431848-22676-0/minicluster-data/ts-1/wal/wals/36e93b81911e4fd0be232dd0ca8cf9cb
I20260620 18:40:32.089999 8940 ts_tablet_manager.cc:1939] T 36e93b81911e4fd0be232dd0ca8cf9cb P fda09ac9343941d1aba5c6e6d16272e1: Deleting consensus metadata
I20260620 18:40:32.091600 8427 catalog_manager.cc:4841] TS fda09ac9343941d1aba5c6e6d16272e1 (127.22.37.2:45967): tablet 36e93b81911e4fd0be232dd0ca8cf9cb (table test-table [id=5009dbc1c2e7481daf78e77582027392]) successfully deleted
I20260620 18:40:32.095161 8916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:32.061458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32842 (local address 127.22.37.1:44709)
0620 18:40:32.061680 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:32.061689 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:32.076198 (+ 14509us) server_negotiation.cc:407] Connection header received
0620 18:40:32.076652 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:32.076657 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:32.076753 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:32.076850 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:40:32.078697 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:32.079809 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:32.081813 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:32.082448 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:32.082550 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:32.088182 (+ 5632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:40:32.088200 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:40:32.088219 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:40:32.088276 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:40:32.092223 (+ 3947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:40:32.092960 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:40:32.092970 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:40:32.092974 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:40:32.093129 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:40:32.093534 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:40:32.093542 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:40:32.093546 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:40:32.094024 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:40:32.094336 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:32.094545 (+ 209us) server_negotiation.cc:299] Negotiation successful
0620 18:40:32.094783 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 18:40:32.095937 8627 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:40:32.202673 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 8504
I20260620 18:40:32.221022 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 8638
Jun 20 18:40:32 dist-test-slave-5lgp krb5kdc[8398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980829, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.37.2@KRBTEST.COM
I20260620 18:40:32.238456 8913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:32.221963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45356 (local address 127.22.37.2:45967)
0620 18:40:32.222212 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:32.222218 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:32.222241 (+ 23us) server_negotiation.cc:407] Connection header received
0620 18:40:32.222308 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:32.222314 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:32.222395 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:32.222506 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:40:32.223841 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:32.225014 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:32.226596 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:32.227221 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:32.227351 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:32.231800 (+ 4449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:40:32.231820 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:40:32.231843 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:40:32.231896 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:40:32.235580 (+ 3684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:40:32.236318 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:40:32.236328 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:40:32.236332 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:40:32.236486 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:40:32.236826 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:40:32.236833 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:40:32.236837 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:40:32.237319 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:40:32.237664 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:32.237851 (+ 187us) server_negotiation.cc:299] Negotiation successful
0620 18:40:32.238083 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260620 18:40:32.239265 8761 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:40:32.330787 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 8638
I20260620 18:40:32.355172 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 8771
Jun 20 18:40:32 dist-test-slave-5lgp krb5kdc[8398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980829, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.37.3@KRBTEST.COM
I20260620 18:40:32.374835 8914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:32.356159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38646 (local address 127.22.37.3:36357)
0620 18:40:32.356330 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:32.356334 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:32.356352 (+ 18us) server_negotiation.cc:407] Connection header received
0620 18:40:32.356410 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:32.356415 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:32.356482 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:32.356575 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:40:32.358459 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:32.359318 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:32.361199 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:32.361791 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:32.361891 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:32.367426 (+ 5535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:40:32.367442 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:40:32.367464 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:40:32.367523 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:40:32.371770 (+ 4247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:40:32.372512 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:40:32.372530 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:40:32.372535 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:40:32.372748 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:40:32.373121 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:40:32.373127 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:40:32.373132 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:40:32.373665 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:40:32.374011 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:32.374169 (+ 158us) server_negotiation.cc:299] Negotiation successful
0620 18:40:32.374395 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 18:40:32.375643 8894 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:40:32.495707 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 8771
I20260620 18:40:32.515403 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 8411
I20260620 18:40:32.516496 8473 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:40:32.626770 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 8411
2026-06-20T18:40:32Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4255 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-20T18:40:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:40:32Z Disabled control of system clock
I20260620 18:40:32.704741 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980527431848-22676-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980527431848-22676-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980527431848-22676-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.62:45659
--webserver_interface=127.22.37.62
--webserver_port=0
--builtin_ntp_servers=127.22.37.20:45395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.37.62:45659
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 18:40:33.033639 8951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:40:33.033921 8951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:40:33.034003 8951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:40:33.041268 8951 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:40:33.041391 8951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:40:33.041476 8951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:40:33.041540 8951 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:40:33.041584 8951 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:40:33.050673 8951 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:45395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980527431848-22676-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980527431848-22676-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.37.62:45659
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.62:45659
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.37.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980527431848-22676-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:40:33.051988 8951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:40:33.053859 8951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:40:33.066179 8956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:40:33.066071 8957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:40:33.066107 8959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:40:33.067145 8951 server_base.cc:1034] running on GCE node
I20260620 18:40:33.068001 8951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:40:33.069411 8951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:40:33.070627 8951 hybrid_clock.cc:648] HybridClock initialized: now 1781980833070580 us; error 46 us; skew 500 ppm
I20260620 18:40:33.070981 8951 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:40:33.073176 8951 webserver.cc:460] Webserver started at http://127.22.37.62:44183/ using document root <none> and password file <none>
I20260620 18:40:33.073918 8951 fs_manager.cc:359] Metadata directory not provided
I20260620 18:40:33.074033 8951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:40:33.074383 8951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:40:33.077244 8951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980527431848-22676-0/minicluster-data/master-0/data/instance:
uuid: "40903511f4984760a10503c1c746c588"
format_stamp: "Formatted at 2026-06-20 18:40:33 on dist-test-slave-5lgp"
server_key: "407f235039127fd18016022e156a1e1e"
server_key_iv: "1d69833e4f8cc7cf11c92c55591ac758"
server_key_version: "encryptionkey@0"
I20260620 18:40:33.078092 8951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980527431848-22676-0/minicluster-data/master-0/wal/instance:
uuid: "40903511f4984760a10503c1c746c588"
format_stamp: "Formatted at 2026-06-20 18:40:33 on dist-test-slave-5lgp"
server_key: "407f235039127fd18016022e156a1e1e"
server_key_iv: "1d69833e4f8cc7cf11c92c55591ac758"
server_key_version: "encryptionkey@0"
I20260620 18:40:33.082998 8951 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 18:40:33.086306 8965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:40:33.087738 8951 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 18:40:33.087929 8951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980527431848-22676-0/minicluster-data/master-0/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980527431848-22676-0/minicluster-data/master-0/wal
uuid: "40903511f4984760a10503c1c746c588"
format_stamp: "Formatted at 2026-06-20 18:40:33 on dist-test-slave-5lgp"
server_key: "407f235039127fd18016022e156a1e1e"
server_key_iv: "1d69833e4f8cc7cf11c92c55591ac758"
server_key_version: "encryptionkey@0"
I20260620 18:40:33.088136 8951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980527431848-22676-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980527431848-22676-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980527431848-22676-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:40:33.106037 8951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:40:33.106986 8951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:40:33.107236 8951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:40:33.125577 8951 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.62:45659
I20260620 18:40:33.125622 9016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.62:45659 every 8 connection(s)
I20260620 18:40:33.126962 8951 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
I20260620 18:40:33.128582 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 8951
I20260620 18:40:33.128769 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980527431848-22676-0/minicluster-data/master-0/wal/instance
I20260620 18:40:33.129273 22676 external_mini_cluster.cc:1428] Setting key 6a55097a133855fbaa3c28043f403434
I20260620 18:40:33.131249 9017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:40:33.146335 9017 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40903511f4984760a10503c1c746c588: Bootstrap starting.
I20260620 18:40:33.147112 9020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:33.131165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42840 (local address 127.22.37.62:45659)
0620 18:40:33.132331 (+ 1166us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:33.132345 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:33.132393 (+ 48us) server_negotiation.cc:407] Connection header received
0620 18:40:33.134483 (+ 2090us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:33.134509 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:33.135109 (+ 600us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:33.135525 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:40:33.136056 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:33.138108 (+ 2052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:33.141258 (+ 3150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:33.142404 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:33.142616 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:33.143888 (+ 1272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:40:33.143900 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:40:33.143919 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:40:33.144703 (+ 784us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:40:33.144805 (+ 102us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:40:33.144832 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:40:33.144937 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:33.145604 (+ 667us) server_negotiation.cc:299] Negotiation successful
0620 18:40:33.145927 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":641,"thread_start_us":417,"threads_started":1}
I20260620 18:40:33.149959 9017 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 40903511f4984760a10503c1c746c588: Neither blocks nor log segments found. Creating new log.
I20260620 18:40:33.151022 9017 log.cc:824] T 00000000000000000000000000000000 P 40903511f4984760a10503c1c746c588: Log is configured to *not* fsync() on all Append() calls
I20260620 18:40:33.153803 9017 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40903511f4984760a10503c1c746c588: No bootstrap required, opened a new log
I20260620 18:40:33.161576 9017 raft_consensus.cc:348] T 00000000000000000000000000000000 P 40903511f4984760a10503c1c746c588 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40903511f4984760a10503c1c746c588" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 45659 } }
I20260620 18:40:33.162040 9017 raft_consensus.cc:374] T 00000000000000000000000000000000 P 40903511f4984760a10503c1c746c588 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:40:33.162148 9017 raft_consensus.cc:729] T 00000000000000000000000000000000 P 40903511f4984760a10503c1c746c588 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40903511f4984760a10503c1c746c588, State: Initialized, Role: FOLLOWER
I20260620 18:40:33.162662 9017 consensus_queue.cc:260] T 00000000000000000000000000000000 P 40903511f4984760a10503c1c746c588 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40903511f4984760a10503c1c746c588" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 45659 } }
I20260620 18:40:33.162860 9017 raft_consensus.cc:388] T 00000000000000000000000000000000 P 40903511f4984760a10503c1c746c588 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:40:33.162963 9017 raft_consensus.cc:482] T 00000000000000000000000000000000 P 40903511f4984760a10503c1c746c588 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:40:33.163087 9017 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 40903511f4984760a10503c1c746c588 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:40:33.165295 9017 raft_consensus.cc:504] T 00000000000000000000000000000000 P 40903511f4984760a10503c1c746c588 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40903511f4984760a10503c1c746c588" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 45659 } }
I20260620 18:40:33.165906 9017 leader_election.cc:302] T 00000000000000000000000000000000 P 40903511f4984760a10503c1c746c588 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 40903511f4984760a10503c1c746c588; no voters:
I20260620 18:40:33.166550 9017 leader_election.cc:288] T 00000000000000000000000000000000 P 40903511f4984760a10503c1c746c588 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:40:33.166697 9022 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 40903511f4984760a10503c1c746c588 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:40:33.167404 9022 raft_consensus.cc:686] T 00000000000000000000000000000000 P 40903511f4984760a10503c1c746c588 [term 1 LEADER]: Becoming Leader. State: Replica: 40903511f4984760a10503c1c746c588, State: Running, Role: LEADER
I20260620 18:40:33.167902 9022 consensus_queue.cc:237] T 00000000000000000000000000000000 P 40903511f4984760a10503c1c746c588 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40903511f4984760a10503c1c746c588" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 45659 } }
I20260620 18:40:33.168715 9017 sys_catalog.cc:564] T 00000000000000000000000000000000 P 40903511f4984760a10503c1c746c588 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:40:33.170617 9024 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40903511f4984760a10503c1c746c588 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40903511f4984760a10503c1c746c588. Latest consensus state: current_term: 1 leader_uuid: "40903511f4984760a10503c1c746c588" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40903511f4984760a10503c1c746c588" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 45659 } } }
I20260620 18:40:33.170943 9024 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40903511f4984760a10503c1c746c588 [sys.catalog]: This master's current role is: LEADER
I20260620 18:40:33.171801 9028 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:40:33.171669 9023 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40903511f4984760a10503c1c746c588 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "40903511f4984760a10503c1c746c588" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40903511f4984760a10503c1c746c588" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 45659 } } }
I20260620 18:40:33.171953 9023 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40903511f4984760a10503c1c746c588 [sys.catalog]: This master's current role is: LEADER
I20260620 18:40:33.176928 9028 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:40:33.186044 9028 catalog_manager.cc:1347] Generated new cluster ID: 6b5e5557633244d48b87bc80ddc42fb9
I20260620 18:40:33.186172 9028 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:40:33.213186 9028 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:40:33.215202 9028 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:40:33.234251 9028 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 40903511f4984760a10503c1c746c588: Generated new TSK 0
I20260620 18:40:33.235333 9028 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:40:33.243717 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980527431848-22676-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980527431848-22676-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980527431848-22676-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980527431848-22676-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.37.1:0
--local_ip_for_outbound_sockets=127.22.37.1
--webserver_interface=127.22.37.1
--webserver_port=0
--tserver_master_addrs=127.22.37.62:45659
--builtin_ntp_servers=127.22.37.20:45395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 18:40:33.571990 9041 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:40:33.572273 9041 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:40:33.572359 9041 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:40:33.579643 9041 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:40:33.579773 9041 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:40:33.579952 9041 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.37.1
I20260620 18:40:33.589161 9041 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:45395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980527431848-22676-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980527431848-22676-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.37.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980527431848-22676-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.37.1
--webserver_port=0
--tserver_master_addrs=127.22.37.62:45659
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.37.1
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980527431848-22676-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:40:33.590456 9041 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:40:33.592294 9041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:40:33.605611 9049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:40:33.605683 9047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:40:33.605638 9046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:40:33.610898 9041 server_base.cc:1034] running on GCE node
I20260620 18:40:33.611665 9041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:40:33.612666 9041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:40:33.613906 9041 hybrid_clock.cc:648] HybridClock initialized: now 1781980833613863 us; error 53 us; skew 500 ppm
I20260620 18:40:33.614252 9041 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:40:33.616649 9041 webserver.cc:460] Webserver started at http://127.22.37.1:34929/ using document root <none> and password file <none>
I20260620 18:40:33.617437 9041 fs_manager.cc:359] Metadata directory not provided
I20260620 18:40:33.617592 9041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:40:33.617920 9041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:40:33.620900 9041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980527431848-22676-0/minicluster-data/ts-0/data/instance:
uuid: "63e3882aa0d1492c9f8aa623f2e4ce1d"
format_stamp: "Formatted at 2026-06-20 18:40:33 on dist-test-slave-5lgp"
server_key: "e11228c490abb2dfbbca1aadc9bc5c4f"
server_key_iv: "bb69b00f183a35ed859864cbb7024a2f"
server_key_version: "encryptionkey@0"
I20260620 18:40:33.621773 9041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980527431848-22676-0/minicluster-data/ts-0/wal/instance:
uuid: "63e3882aa0d1492c9f8aa623f2e4ce1d"
format_stamp: "Formatted at 2026-06-20 18:40:33 on dist-test-slave-5lgp"
server_key: "e11228c490abb2dfbbca1aadc9bc5c4f"
server_key_iv: "bb69b00f183a35ed859864cbb7024a2f"
server_key_version: "encryptionkey@0"
I20260620 18:40:33.627225 9041 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260620 18:40:33.630622 9055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:40:33.631933 9041 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:40:33.632119 9041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980527431848-22676-0/minicluster-data/ts-0/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980527431848-22676-0/minicluster-data/ts-0/wal
uuid: "63e3882aa0d1492c9f8aa623f2e4ce1d"
format_stamp: "Formatted at 2026-06-20 18:40:33 on dist-test-slave-5lgp"
server_key: "e11228c490abb2dfbbca1aadc9bc5c4f"
server_key_iv: "bb69b00f183a35ed859864cbb7024a2f"
server_key_version: "encryptionkey@0"
I20260620 18:40:33.632352 9041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980527431848-22676-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980527431848-22676-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980527431848-22676-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:40:33.650061 9041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:40:33.650923 9041 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:40:33.651227 9041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:40:33.652244 9041 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:40:33.653676 9041 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:40:33.653807 9041 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:40:33.653946 9041 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:40:33.654045 9041 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:40:33.687242 9041 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.1:35735
I20260620 18:40:33.687296 9167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.1:35735 every 8 connection(s)
I20260620 18:40:33.688704 9041 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980527431848-22676-0/minicluster-data/ts-0/data/info.pb
I20260620 18:40:33.697995 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 9041
I20260620 18:40:33.698187 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980527431848-22676-0/minicluster-data/ts-0/wal/instance
I20260620 18:40:33.698761 22676 external_mini_cluster.cc:1428] Setting key cb3802eeba8198f591e03087e3967665
I20260620 18:40:33.701911 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980527431848-22676-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980527431848-22676-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980527431848-22676-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980527431848-22676-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.37.2:0
--local_ip_for_outbound_sockets=127.22.37.2
--webserver_interface=127.22.37.2
--webserver_port=0
--tserver_master_addrs=127.22.37.62:45659
--builtin_ntp_servers=127.22.37.20:45395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 18:40:33.702489 9170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:33.691468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.1:34145 (local address 127.22.37.62:45659)
0620 18:40:33.691956 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:33.691961 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:33.693318 (+ 1357us) server_negotiation.cc:407] Connection header received
0620 18:40:33.695192 (+ 1874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:33.695198 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:33.695291 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:33.695413 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:40:33.696480 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:33.697232 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:33.699582 (+ 2350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:33.700016 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:33.700155 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:33.701635 (+ 1480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:40:33.701640 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:40:33.701644 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:40:33.701661 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:40:33.701700 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:40:33.701713 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:40:33.701788 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:33.702119 (+ 331us) server_negotiation.cc:299] Negotiation successful
0620 18:40:33.702222 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":280,"threads_started":1}
I20260620 18:40:33.703930 9171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:33.692146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:45659 (local address 127.22.37.1:34145)
0620 18:40:33.693160 (+ 1014us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:33.693202 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 18:40:33.694910 (+ 1708us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:40:33.695649 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:40:33.695658 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:40:33.695747 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:40:33.696315 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:33.696331 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:33.697411 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:33.697417 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:33.699436 (+ 2019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:33.699449 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:33.700237 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:33.700247 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:33.700498 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:33.701232 (+ 734us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:40:33.701235 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:40:33.701286 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:40:33.701422 (+ 136us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:40:33.701820 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:40:33.701832 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:40:33.701841 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 18:40:33.702053 (+ 212us) client_negotiation.cc:240] Negotiation successful
0620 18:40:33.702494 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":273,"threads_started":1}
I20260620 18:40:33.705567 9168 heartbeater.cc:347] Connected to a master server at 127.22.37.62:45659
I20260620 18:40:33.705883 9168 heartbeater.cc:464] Registering TS with master...
I20260620 18:40:33.706650 9168 heartbeater.cc:511] Master 127.22.37.62:45659 requested a full tablet report, sending...
I20260620 18:40:33.708814 8982 ts_manager.cc:194] Registered new tserver with Master: 63e3882aa0d1492c9f8aa623f2e4ce1d (127.22.37.1:35735)
I20260620 18:40:33.710978 8982 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.37.1:34145
W20260620 18:40:34.033879 9172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:40:34.034168 9172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:40:34.034334 9172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:40:34.041726 9172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:40:34.041826 9172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:40:34.042022 9172 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.37.2
I20260620 18:40:34.051164 9172 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:45395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980527431848-22676-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980527431848-22676-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.37.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980527431848-22676-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.37.2
--webserver_port=0
--tserver_master_addrs=127.22.37.62:45659
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.37.2
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980527431848-22676-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:40:34.052512 9172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:40:34.054319 9172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:40:34.067227 9180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:40:34.067236 9178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:40:34.067584 9177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:40:34.068126 9172 server_base.cc:1034] running on GCE node
I20260620 18:40:34.068725 9172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:40:34.069751 9172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:40:34.070969 9172 hybrid_clock.cc:648] HybridClock initialized: now 1781980834070928 us; error 52 us; skew 500 ppm
I20260620 18:40:34.071273 9172 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:40:34.073639 9172 webserver.cc:460] Webserver started at http://127.22.37.2:46123/ using document root <none> and password file <none>
I20260620 18:40:34.074331 9172 fs_manager.cc:359] Metadata directory not provided
I20260620 18:40:34.074460 9172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:40:34.074765 9172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:40:34.077608 9172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980527431848-22676-0/minicluster-data/ts-1/data/instance:
uuid: "fc6d70339f694cfd8b046842726a0d66"
format_stamp: "Formatted at 2026-06-20 18:40:34 on dist-test-slave-5lgp"
server_key: "2ed60d85e25e59eff29c74582329ca94"
server_key_iv: "ad401a0dcd2e6e5d5df0e445c8eafe79"
server_key_version: "encryptionkey@0"
I20260620 18:40:34.078382 9172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980527431848-22676-0/minicluster-data/ts-1/wal/instance:
uuid: "fc6d70339f694cfd8b046842726a0d66"
format_stamp: "Formatted at 2026-06-20 18:40:34 on dist-test-slave-5lgp"
server_key: "2ed60d85e25e59eff29c74582329ca94"
server_key_iv: "ad401a0dcd2e6e5d5df0e445c8eafe79"
server_key_version: "encryptionkey@0"
I20260620 18:40:34.083230 9172 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260620 18:40:34.086519 9186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:40:34.087680 9172 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s
I20260620 18:40:34.087848 9172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980527431848-22676-0/minicluster-data/ts-1/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980527431848-22676-0/minicluster-data/ts-1/wal
uuid: "fc6d70339f694cfd8b046842726a0d66"
format_stamp: "Formatted at 2026-06-20 18:40:34 on dist-test-slave-5lgp"
server_key: "2ed60d85e25e59eff29c74582329ca94"
server_key_iv: "ad401a0dcd2e6e5d5df0e445c8eafe79"
server_key_version: "encryptionkey@0"
I20260620 18:40:34.088043 9172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980527431848-22676-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980527431848-22676-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980527431848-22676-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:40:34.108777 9172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:40:34.109742 9172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:40:34.110033 9172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:40:34.110975 9172 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:40:34.112381 9172 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:40:34.112500 9172 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:40:34.112634 9172 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:40:34.112756 9172 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:40:34.151889 9172 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.2:42033
I20260620 18:40:34.151978 9298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.2:42033 every 8 connection(s)
I20260620 18:40:34.153218 9172 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980527431848-22676-0/minicluster-data/ts-1/data/info.pb
I20260620 18:40:34.159799 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 9172
I20260620 18:40:34.159991 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980527431848-22676-0/minicluster-data/ts-1/wal/instance
I20260620 18:40:34.160552 22676 external_mini_cluster.cc:1428] Setting key 04fc27afc87473c5d8b65e720903e0be
I20260620 18:40:34.163734 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980527431848-22676-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980527431848-22676-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980527431848-22676-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980527431848-22676-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.37.3:0
--local_ip_for_outbound_sockets=127.22.37.3
--webserver_interface=127.22.37.3
--webserver_port=0
--tserver_master_addrs=127.22.37.62:45659
--builtin_ntp_servers=127.22.37.20:45395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 18:40:34.166859 9170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:34.155866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.2:60405 (local address 127.22.37.62:45659)
0620 18:40:34.156047 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:34.156052 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:34.157512 (+ 1460us) server_negotiation.cc:407] Connection header received
0620 18:40:34.159313 (+ 1801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:34.159318 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:34.159394 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:34.159500 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:40:34.160937 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:34.161918 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:34.164074 (+ 2156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:34.164447 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:34.164544 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:34.166174 (+ 1630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:40:34.166179 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:40:34.166182 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:40:34.166199 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:40:34.166230 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:40:34.166242 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:40:34.166312 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:34.166564 (+ 252us) server_negotiation.cc:299] Negotiation successful
0620 18:40:34.166638 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 18:40:34.168355 9301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:34.156396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:45659 (local address 127.22.37.2:60405)
0620 18:40:34.157306 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:34.157343 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 18:40:34.159024 (+ 1681us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:40:34.159796 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:40:34.159807 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:40:34.159942 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:40:34.160763 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:34.160788 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:34.162062 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:34.162066 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:34.163924 (+ 1858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:34.163942 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:34.164683 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:34.164693 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:34.165039 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:34.165805 (+ 766us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:40:34.165811 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:40:34.165865 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:40:34.165979 (+ 114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:40:34.166339 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:40:34.166353 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:40:34.166363 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 18:40:34.166542 (+ 179us) client_negotiation.cc:240] Negotiation successful
0620 18:40:34.166964 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":276,"threads_started":1}
I20260620 18:40:34.170006 9299 heartbeater.cc:347] Connected to a master server at 127.22.37.62:45659
I20260620 18:40:34.170322 9299 heartbeater.cc:464] Registering TS with master...
I20260620 18:40:34.171227 9299 heartbeater.cc:511] Master 127.22.37.62:45659 requested a full tablet report, sending...
I20260620 18:40:34.172807 8982 ts_manager.cc:194] Registered new tserver with Master: fc6d70339f694cfd8b046842726a0d66 (127.22.37.2:42033)
I20260620 18:40:34.173695 8982 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.37.2:60405
W20260620 18:40:34.491727 9302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:40:34.491997 9302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:40:34.492178 9302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:40:34.499439 9302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:40:34.499537 9302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:40:34.499644 9302 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.37.3
I20260620 18:40:34.508584 9302 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:45395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980527431848-22676-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980527431848-22676-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.37.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980527431848-22676-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.37.3
--webserver_port=0
--tserver_master_addrs=127.22.37.62:45659
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.37.3
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980527431848-22676-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:40:34.509945 9302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:40:34.511757 9302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:40:34.524636 9310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:40:34.524732 9307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:40:34.524693 9308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:40:34.525538 9302 server_base.cc:1034] running on GCE node
I20260620 18:40:34.526234 9302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:40:34.527215 9302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:40:34.528439 9302 hybrid_clock.cc:648] HybridClock initialized: now 1781980834528387 us; error 68 us; skew 500 ppm
I20260620 18:40:34.528753 9302 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:40:34.530949 9302 webserver.cc:460] Webserver started at http://127.22.37.3:36053/ using document root <none> and password file <none>
I20260620 18:40:34.531628 9302 fs_manager.cc:359] Metadata directory not provided
I20260620 18:40:34.531754 9302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:40:34.532051 9302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:40:34.534955 9302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980527431848-22676-0/minicluster-data/ts-2/data/instance:
uuid: "08cf15f24c4d4e638e5917ecd55eb90a"
format_stamp: "Formatted at 2026-06-20 18:40:34 on dist-test-slave-5lgp"
server_key: "45c87e37ec25dcac222ba1a5de677e7f"
server_key_iv: "8a95b370570ce0f22de12aad586008c9"
server_key_version: "encryptionkey@0"
I20260620 18:40:34.535811 9302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980527431848-22676-0/minicluster-data/ts-2/wal/instance:
uuid: "08cf15f24c4d4e638e5917ecd55eb90a"
format_stamp: "Formatted at 2026-06-20 18:40:34 on dist-test-slave-5lgp"
server_key: "45c87e37ec25dcac222ba1a5de677e7f"
server_key_iv: "8a95b370570ce0f22de12aad586008c9"
server_key_version: "encryptionkey@0"
I20260620 18:40:34.540920 9302 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.006s
I20260620 18:40:34.544258 9316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:40:34.545611 9302 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 18:40:34.545771 9302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980527431848-22676-0/minicluster-data/ts-2/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980527431848-22676-0/minicluster-data/ts-2/wal
uuid: "08cf15f24c4d4e638e5917ecd55eb90a"
format_stamp: "Formatted at 2026-06-20 18:40:34 on dist-test-slave-5lgp"
server_key: "45c87e37ec25dcac222ba1a5de677e7f"
server_key_iv: "8a95b370570ce0f22de12aad586008c9"
server_key_version: "encryptionkey@0"
I20260620 18:40:34.545975 9302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980527431848-22676-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980527431848-22676-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980527431848-22676-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:40:34.571066 9302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:40:34.571943 9302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:40:34.572233 9302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:40:34.573282 9302 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:40:34.574755 9302 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:40:34.574875 9302 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:40:34.575006 9302 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:40:34.575106 9302 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:40:34.608173 9302 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.3:37183
I20260620 18:40:34.608263 9428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.3:37183 every 8 connection(s)
I20260620 18:40:34.609616 9302 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980527431848-22676-0/minicluster-data/ts-2/data/info.pb
I20260620 18:40:34.611399 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 9302
I20260620 18:40:34.611573 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980527431848-22676-0/minicluster-data/ts-2/wal/instance
I20260620 18:40:34.612139 22676 external_mini_cluster.cc:1428] Setting key 6fe2541dc60ff68608018b8ff44d5455
I20260620 18:40:34.622632 9170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:34.612458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.3:41657 (local address 127.22.37.62:45659)
0620 18:40:34.612680 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:34.612686 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:34.614465 (+ 1779us) server_negotiation.cc:407] Connection header received
0620 18:40:34.616530 (+ 2065us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:34.616534 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:34.616609 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:34.616695 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:40:34.617807 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:34.618550 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:34.620026 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:34.620429 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:34.620519 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:34.621947 (+ 1428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:40:34.621952 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:40:34.621955 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:40:34.621973 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:40:34.622006 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:40:34.622018 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:40:34.622095 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:34.622298 (+ 203us) server_negotiation.cc:299] Negotiation successful
0620 18:40:34.622378 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 18:40:34.623652 9431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:34.613094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:45659 (local address 127.22.37.3:41657)
0620 18:40:34.614271 (+ 1177us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:34.614344 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0620 18:40:34.616275 (+ 1931us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:40:34.616901 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:40:34.616909 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:40:34.617008 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:40:34.617654 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:34.617672 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:34.618675 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:34.618679 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:34.619899 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:34.619911 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:34.620625 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:34.620629 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:34.620848 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:34.621635 (+ 787us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:40:34.621638 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:40:34.621678 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:40:34.621762 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:40:34.622106 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:40:34.622113 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:40:34.622119 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 18:40:34.622256 (+ 137us) client_negotiation.cc:240] Negotiation successful
0620 18:40:34.622559 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":386,"threads_started":1}
I20260620 18:40:34.625059 9429 heartbeater.cc:347] Connected to a master server at 127.22.37.62:45659
I20260620 18:40:34.625432 9429 heartbeater.cc:464] Registering TS with master...
I20260620 18:40:34.626356 9429 heartbeater.cc:511] Master 127.22.37.62:45659 requested a full tablet report, sending...
I20260620 18:40:34.627970 8982 ts_manager.cc:194] Registered new tserver with Master: 08cf15f24c4d4e638e5917ecd55eb90a (127.22.37.3:37183)
I20260620 18:40:34.628784 8982 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.37.3:41657
I20260620 18:40:34.640340 22676 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:40:34.650676 9170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:34.644700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42850 (local address 127.22.37.62:45659)
0620 18:40:34.644911 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:34.644921 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:34.645363 (+ 442us) server_negotiation.cc:407] Connection header received
0620 18:40:34.645586 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:34.645590 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:34.645677 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:34.645760 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:40:34.646255 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:34.646975 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:34.648295 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:34.648733 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:34.648826 (+ 93us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:34.649950 (+ 1124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:40:34.649955 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:40:34.649958 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:40:34.649975 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:40:34.650006 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:40:34.650017 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:40:34.650207 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:34.650351 (+ 144us) server_negotiation.cc:299] Negotiation successful
0620 18:40:34.650394 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 18:40:34.654614 8982 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42850:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:40:34.661635 8982 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:40:34.700176 9443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:34.674795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.1:35735 (local address 127.0.0.1:60894)
0620 18:40:34.688620 (+ 13825us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:34.688639 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:40:34.688891 (+ 252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:40:34.690221 (+ 1330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:40:34.690233 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:40:34.690281 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:40:34.690811 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:34.690828 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:34.693764 (+ 2936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:34.693772 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:34.696837 (+ 3065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:34.696867 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:34.699028 (+ 2161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:34.699039 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:34.699306 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:34.699370 (+ 64us) client_negotiation.cc:769] Sending connection context
0620 18:40:34.699513 (+ 143us) client_negotiation.cc:240] Negotiation successful
0620 18:40:34.699723 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13696,"mutex_wait_us":3,"thread_start_us":13562,"threads_started":1}
I20260620 18:40:34.700946 9441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:34.674315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.3:37183 (local address 127.0.0.1:45526)
0620 18:40:34.687744 (+ 13429us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:34.687796 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 18:40:34.688073 (+ 277us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:40:34.692743 (+ 4670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:40:34.692750 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:40:34.692789 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:40:34.693195 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:34.693209 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:34.695557 (+ 2348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:34.695563 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:34.697647 (+ 2084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:34.697665 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:34.699662 (+ 1997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:34.699668 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:34.699855 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:34.700326 (+ 471us) client_negotiation.cc:769] Sending connection context
0620 18:40:34.700416 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 18:40:34.700555 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13104,"mutex_wait_us":5,"thread_start_us":12936,"threads_started":1}
I20260620 18:40:34.700990 9444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:34.675440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60894 (local address 127.22.37.1:35735)
0620 18:40:34.686719 (+ 11279us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:34.686733 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:34.688746 (+ 2013us) server_negotiation.cc:407] Connection header received
0620 18:40:34.689119 (+ 373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:34.689128 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:34.689491 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:34.689726 (+ 235us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:40:34.691061 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:34.693516 (+ 2455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:34.697109 (+ 3593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:34.698758 (+ 1649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:34.700020 (+ 1262us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:34.700330 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:34.700414 (+ 84us) server_negotiation.cc:299] Negotiation successful
0620 18:40:34.700587 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11069,"thread_start_us":11374,"threads_started":1}
I20260620 18:40:34.701809 9442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:34.674656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45526 (local address 127.22.37.3:37183)
0620 18:40:34.691729 (+ 17073us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:34.691738 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:34.691782 (+ 44us) server_negotiation.cc:407] Connection header received
0620 18:40:34.691894 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:34.691918 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:34.692197 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:34.693447 (+ 1250us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:40:34.693754 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:34.695371 (+ 1617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:34.697982 (+ 2611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:34.699450 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:34.699597 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:34.699876 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:34.701206 (+ 1330us) server_negotiation.cc:299] Negotiation successful
0620 18:40:34.701414 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16848,"thread_start_us":17616,"threads_started":1}
I20260620 18:40:34.705646 9446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:34.695283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.2:42033 (local address 127.0.0.1:42450)
0620 18:40:34.697576 (+ 2293us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:34.697596 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:40:34.697723 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:40:34.698745 (+ 1022us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:40:34.698749 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:40:34.698778 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:40:34.699170 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:34.699179 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:34.701346 (+ 2167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:34.701350 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:34.703407 (+ 2057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:34.703420 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:34.704758 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:34.704765 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:34.704949 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:34.704985 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 18:40:34.705093 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 18:40:34.705234 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2133,"mutex_wait_us":60,"thread_start_us":1957,"threads_started":1}
I20260620 18:40:34.705736 9445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:34.689124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42450 (local address 127.22.37.2:42033)
0620 18:40:34.698216 (+ 9092us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:34.698224 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:34.698258 (+ 34us) server_negotiation.cc:407] Connection header received
0620 18:40:34.698355 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:34.698361 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:34.698552 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:34.698764 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:40:34.699425 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:34.700867 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:34.703608 (+ 2741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:34.704548 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:34.704728 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:34.704989 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:34.705135 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 18:40:34.705308 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8923,"thread_start_us":8798,"threads_started":1}
I20260620 18:40:34.705662 9103 tablet_service.cc:1467] Processing CreateTablet for tablet 7274279c27e54f039592e138a125b179 (DEFAULT_TABLE table=test-table [id=61d0da845ea54eb185b50657e20a7b07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:40:34.705701 9364 tablet_service.cc:1467] Processing CreateTablet for tablet 7274279c27e54f039592e138a125b179 (DEFAULT_TABLE table=test-table [id=61d0da845ea54eb185b50657e20a7b07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:40:34.707276 9103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7274279c27e54f039592e138a125b179. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:40:34.707283 9364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7274279c27e54f039592e138a125b179. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:40:34.709039 9234 tablet_service.cc:1467] Processing CreateTablet for tablet 7274279c27e54f039592e138a125b179 (DEFAULT_TABLE table=test-table [id=61d0da845ea54eb185b50657e20a7b07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:40:34.710266 9234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7274279c27e54f039592e138a125b179. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:40:34.714740 9168 heartbeater.cc:503] Master 127.22.37.62:45659 was elected leader, sending a full tablet report...
I20260620 18:40:34.722893 9447 tablet_bootstrap.cc:492] T 7274279c27e54f039592e138a125b179 P 63e3882aa0d1492c9f8aa623f2e4ce1d: Bootstrap starting.
I20260620 18:40:34.724788 9448 tablet_bootstrap.cc:492] T 7274279c27e54f039592e138a125b179 P fc6d70339f694cfd8b046842726a0d66: Bootstrap starting.
I20260620 18:40:34.725976 9449 tablet_bootstrap.cc:492] T 7274279c27e54f039592e138a125b179 P 08cf15f24c4d4e638e5917ecd55eb90a: Bootstrap starting.
I20260620 18:40:34.728796 9449 tablet_bootstrap.cc:654] T 7274279c27e54f039592e138a125b179 P 08cf15f24c4d4e638e5917ecd55eb90a: Neither blocks nor log segments found. Creating new log.
I20260620 18:40:34.728807 9447 tablet_bootstrap.cc:654] T 7274279c27e54f039592e138a125b179 P 63e3882aa0d1492c9f8aa623f2e4ce1d: Neither blocks nor log segments found. Creating new log.
I20260620 18:40:34.729893 9449 log.cc:824] T 7274279c27e54f039592e138a125b179 P 08cf15f24c4d4e638e5917ecd55eb90a: Log is configured to *not* fsync() on all Append() calls
I20260620 18:40:34.730338 9447 log.cc:824] T 7274279c27e54f039592e138a125b179 P 63e3882aa0d1492c9f8aa623f2e4ce1d: Log is configured to *not* fsync() on all Append() calls
I20260620 18:40:34.731739 9448 tablet_bootstrap.cc:654] T 7274279c27e54f039592e138a125b179 P fc6d70339f694cfd8b046842726a0d66: Neither blocks nor log segments found. Creating new log.
I20260620 18:40:34.732836 9448 log.cc:824] T 7274279c27e54f039592e138a125b179 P fc6d70339f694cfd8b046842726a0d66: Log is configured to *not* fsync() on all Append() calls
I20260620 18:40:34.733577 9449 tablet_bootstrap.cc:492] T 7274279c27e54f039592e138a125b179 P 08cf15f24c4d4e638e5917ecd55eb90a: No bootstrap required, opened a new log
I20260620 18:40:34.733862 9449 ts_tablet_manager.cc:1397] T 7274279c27e54f039592e138a125b179 P 08cf15f24c4d4e638e5917ecd55eb90a: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.002s
I20260620 18:40:34.734344 9447 tablet_bootstrap.cc:492] T 7274279c27e54f039592e138a125b179 P 63e3882aa0d1492c9f8aa623f2e4ce1d: No bootstrap required, opened a new log
I20260620 18:40:34.734623 9447 ts_tablet_manager.cc:1397] T 7274279c27e54f039592e138a125b179 P 63e3882aa0d1492c9f8aa623f2e4ce1d: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260620 18:40:34.735783 9448 tablet_bootstrap.cc:492] T 7274279c27e54f039592e138a125b179 P fc6d70339f694cfd8b046842726a0d66: No bootstrap required, opened a new log
I20260620 18:40:34.736058 9448 ts_tablet_manager.cc:1397] T 7274279c27e54f039592e138a125b179 P fc6d70339f694cfd8b046842726a0d66: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.002s
I20260620 18:40:34.742381 9449 raft_consensus.cc:348] T 7274279c27e54f039592e138a125b179 P 08cf15f24c4d4e638e5917ecd55eb90a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08cf15f24c4d4e638e5917ecd55eb90a" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 37183 } } peers { permanent_uuid: "63e3882aa0d1492c9f8aa623f2e4ce1d" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 35735 } } peers { permanent_uuid: "fc6d70339f694cfd8b046842726a0d66" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 42033 } }
I20260620 18:40:34.742815 9449 raft_consensus.cc:374] T 7274279c27e54f039592e138a125b179 P 08cf15f24c4d4e638e5917ecd55eb90a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:40:34.742909 9449 raft_consensus.cc:729] T 7274279c27e54f039592e138a125b179 P 08cf15f24c4d4e638e5917ecd55eb90a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08cf15f24c4d4e638e5917ecd55eb90a, State: Initialized, Role: FOLLOWER
I20260620 18:40:34.743567 9449 consensus_queue.cc:260] T 7274279c27e54f039592e138a125b179 P 08cf15f24c4d4e638e5917ecd55eb90a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08cf15f24c4d4e638e5917ecd55eb90a" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 37183 } } peers { permanent_uuid: "63e3882aa0d1492c9f8aa623f2e4ce1d" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 35735 } } peers { permanent_uuid: "fc6d70339f694cfd8b046842726a0d66" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 42033 } }
I20260620 18:40:34.744671 9429 heartbeater.cc:503] Master 127.22.37.62:45659 was elected leader, sending a full tablet report...
I20260620 18:40:34.745445 9449 ts_tablet_manager.cc:1428] T 7274279c27e54f039592e138a125b179 P 08cf15f24c4d4e638e5917ecd55eb90a: Time spent starting tablet: real 0.011s user 0.003s sys 0.008s
I20260620 18:40:34.745308 9447 raft_consensus.cc:348] T 7274279c27e54f039592e138a125b179 P 63e3882aa0d1492c9f8aa623f2e4ce1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08cf15f24c4d4e638e5917ecd55eb90a" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 37183 } } peers { permanent_uuid: "63e3882aa0d1492c9f8aa623f2e4ce1d" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 35735 } } peers { permanent_uuid: "fc6d70339f694cfd8b046842726a0d66" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 42033 } }
I20260620 18:40:34.745787 9447 raft_consensus.cc:374] T 7274279c27e54f039592e138a125b179 P 63e3882aa0d1492c9f8aa623f2e4ce1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:40:34.745888 9447 raft_consensus.cc:729] T 7274279c27e54f039592e138a125b179 P 63e3882aa0d1492c9f8aa623f2e4ce1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63e3882aa0d1492c9f8aa623f2e4ce1d, State: Initialized, Role: FOLLOWER
I20260620 18:40:34.746491 9447 consensus_queue.cc:260] T 7274279c27e54f039592e138a125b179 P 63e3882aa0d1492c9f8aa623f2e4ce1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08cf15f24c4d4e638e5917ecd55eb90a" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 37183 } } peers { permanent_uuid: "63e3882aa0d1492c9f8aa623f2e4ce1d" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 35735 } } peers { permanent_uuid: "fc6d70339f694cfd8b046842726a0d66" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 42033 } }
I20260620 18:40:34.746492 9448 raft_consensus.cc:348] T 7274279c27e54f039592e138a125b179 P fc6d70339f694cfd8b046842726a0d66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08cf15f24c4d4e638e5917ecd55eb90a" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 37183 } } peers { permanent_uuid: "63e3882aa0d1492c9f8aa623f2e4ce1d" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 35735 } } peers { permanent_uuid: "fc6d70339f694cfd8b046842726a0d66" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 42033 } }
I20260620 18:40:34.746882 9448 raft_consensus.cc:374] T 7274279c27e54f039592e138a125b179 P fc6d70339f694cfd8b046842726a0d66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:40:34.746985 9448 raft_consensus.cc:729] T 7274279c27e54f039592e138a125b179 P fc6d70339f694cfd8b046842726a0d66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc6d70339f694cfd8b046842726a0d66, State: Initialized, Role: FOLLOWER
I20260620 18:40:34.747870 9448 consensus_queue.cc:260] T 7274279c27e54f039592e138a125b179 P fc6d70339f694cfd8b046842726a0d66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08cf15f24c4d4e638e5917ecd55eb90a" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 37183 } } peers { permanent_uuid: "63e3882aa0d1492c9f8aa623f2e4ce1d" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 35735 } } peers { permanent_uuid: "fc6d70339f694cfd8b046842726a0d66" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 42033 } }
I20260620 18:40:34.749141 9299 heartbeater.cc:503] Master 127.22.37.62:45659 was elected leader, sending a full tablet report...
I20260620 18:40:34.750195 9448 ts_tablet_manager.cc:1428] T 7274279c27e54f039592e138a125b179 P fc6d70339f694cfd8b046842726a0d66: Time spent starting tablet: real 0.014s user 0.014s sys 0.001s
I20260620 18:40:34.754631 9447 ts_tablet_manager.cc:1428] T 7274279c27e54f039592e138a125b179 P 63e3882aa0d1492c9f8aa623f2e4ce1d: Time spent starting tablet: real 0.020s user 0.020s sys 0.000s
I20260620 18:40:34.798866 9454 raft_consensus.cc:482] T 7274279c27e54f039592e138a125b179 P 63e3882aa0d1492c9f8aa623f2e4ce1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:40:34.799158 9454 raft_consensus.cc:504] T 7274279c27e54f039592e138a125b179 P 63e3882aa0d1492c9f8aa623f2e4ce1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08cf15f24c4d4e638e5917ecd55eb90a" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 37183 } } peers { permanent_uuid: "63e3882aa0d1492c9f8aa623f2e4ce1d" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 35735 } } peers { permanent_uuid: "fc6d70339f694cfd8b046842726a0d66" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 42033 } }
I20260620 18:40:34.800747 9454 leader_election.cc:288] T 7274279c27e54f039592e138a125b179 P 63e3882aa0d1492c9f8aa623f2e4ce1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08cf15f24c4d4e638e5917ecd55eb90a (127.22.37.3:37183), fc6d70339f694cfd8b046842726a0d66 (127.22.37.2:42033)
I20260620 18:40:34.807925 9445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:34.801189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.1:48821 (local address 127.22.37.2:42033)
0620 18:40:34.801427 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:34.801435 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:34.801935 (+ 500us) server_negotiation.cc:407] Connection header received
0620 18:40:34.802188 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:34.802200 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:34.802270 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:34.802318 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:40:34.803187 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:34.803842 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:34.805834 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:34.806948 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:34.807113 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:34.807220 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:34.807513 (+ 293us) server_negotiation.cc:299] Negotiation successful
0620 18:40:34.807664 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 18:40:34.807940 9442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:34.801192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.1:52635 (local address 127.22.37.3:37183)
0620 18:40:34.801426 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:34.801436 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:34.802028 (+ 592us) server_negotiation.cc:407] Connection header received
0620 18:40:34.802198 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:34.802204 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:34.802279 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:34.802374 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:40:34.803141 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:34.803830 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:34.805911 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:34.807115 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:34.807226 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:34.807315 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:34.807602 (+ 287us) server_negotiation.cc:299] Negotiation successful
0620 18:40:34.807717 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 18:40:34.808079 9456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:34.801020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.2:42033 (local address 127.22.37.1:48821)
0620 18:40:34.801802 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:34.801834 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 18:40:34.801993 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:40:34.802545 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:40:34.802550 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:40:34.802587 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:40:34.803070 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:34.803083 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:34.803988 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:34.803994 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:34.805665 (+ 1671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:34.805676 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:34.807148 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:34.807152 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:34.807347 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:34.807385 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 18:40:34.807482 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 18:40:34.807656 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"mutex_wait_us":84,"thread_start_us":420,"threads_started":1}
I20260620 18:40:34.808128 9457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:34.801139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.3:37183 (local address 127.22.37.1:52635)
0620 18:40:34.801943 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:34.801958 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 18:40:34.802056 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:40:34.802412 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:40:34.802417 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:40:34.802454 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:40:34.802937 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:34.802950 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:34.803963 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:34.803971 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:34.805730 (+ 1759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:34.805744 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:34.807287 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:34.807293 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:34.807445 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:34.807483 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 18:40:34.807556 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 18:40:34.807689 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":690,"mutex_wait_us":139,"thread_start_us":478,"threads_started":1}
I20260620 18:40:34.808965 9254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7274279c27e54f039592e138a125b179" candidate_uuid: "63e3882aa0d1492c9f8aa623f2e4ce1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc6d70339f694cfd8b046842726a0d66" is_pre_election: true
I20260620 18:40:34.809311 9384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7274279c27e54f039592e138a125b179" candidate_uuid: "63e3882aa0d1492c9f8aa623f2e4ce1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08cf15f24c4d4e638e5917ecd55eb90a" is_pre_election: true
I20260620 18:40:34.809530 9254 raft_consensus.cc:2454] T 7274279c27e54f039592e138a125b179 P fc6d70339f694cfd8b046842726a0d66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63e3882aa0d1492c9f8aa623f2e4ce1d in term 0.
I20260620 18:40:34.809851 9384 raft_consensus.cc:2454] T 7274279c27e54f039592e138a125b179 P 08cf15f24c4d4e638e5917ecd55eb90a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63e3882aa0d1492c9f8aa623f2e4ce1d in term 0.
I20260620 18:40:34.810218 9056 leader_election.cc:302] T 7274279c27e54f039592e138a125b179 P 63e3882aa0d1492c9f8aa623f2e4ce1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63e3882aa0d1492c9f8aa623f2e4ce1d, fc6d70339f694cfd8b046842726a0d66; no voters:
I20260620 18:40:34.810616 9454 raft_consensus.cc:2781] T 7274279c27e54f039592e138a125b179 P 63e3882aa0d1492c9f8aa623f2e4ce1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:40:34.810786 9454 raft_consensus.cc:482] T 7274279c27e54f039592e138a125b179 P 63e3882aa0d1492c9f8aa623f2e4ce1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:40:34.810891 9454 raft_consensus.cc:3037] T 7274279c27e54f039592e138a125b179 P 63e3882aa0d1492c9f8aa623f2e4ce1d [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:40:34.813076 9454 raft_consensus.cc:504] T 7274279c27e54f039592e138a125b179 P 63e3882aa0d1492c9f8aa623f2e4ce1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08cf15f24c4d4e638e5917ecd55eb90a" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 37183 } } peers { permanent_uuid: "63e3882aa0d1492c9f8aa623f2e4ce1d" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 35735 } } peers { permanent_uuid: "fc6d70339f694cfd8b046842726a0d66" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 42033 } }
I20260620 18:40:34.813759 9454 leader_election.cc:288] T 7274279c27e54f039592e138a125b179 P 63e3882aa0d1492c9f8aa623f2e4ce1d [CANDIDATE]: Term 1 election: Requested vote from peers 08cf15f24c4d4e638e5917ecd55eb90a (127.22.37.3:37183), fc6d70339f694cfd8b046842726a0d66 (127.22.37.2:42033)
I20260620 18:40:34.814124 9384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7274279c27e54f039592e138a125b179" candidate_uuid: "63e3882aa0d1492c9f8aa623f2e4ce1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08cf15f24c4d4e638e5917ecd55eb90a"
I20260620 18:40:34.814158 9254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7274279c27e54f039592e138a125b179" candidate_uuid: "63e3882aa0d1492c9f8aa623f2e4ce1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc6d70339f694cfd8b046842726a0d66"
I20260620 18:40:34.814323 9384 raft_consensus.cc:3037] T 7274279c27e54f039592e138a125b179 P 08cf15f24c4d4e638e5917ecd55eb90a [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:40:34.814344 9254 raft_consensus.cc:3037] T 7274279c27e54f039592e138a125b179 P fc6d70339f694cfd8b046842726a0d66 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:40:34.816442 9384 raft_consensus.cc:2454] T 7274279c27e54f039592e138a125b179 P 08cf15f24c4d4e638e5917ecd55eb90a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63e3882aa0d1492c9f8aa623f2e4ce1d in term 1.
I20260620 18:40:34.816550 9254 raft_consensus.cc:2454] T 7274279c27e54f039592e138a125b179 P fc6d70339f694cfd8b046842726a0d66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63e3882aa0d1492c9f8aa623f2e4ce1d in term 1.
I20260620 18:40:34.816835 9059 leader_election.cc:302] T 7274279c27e54f039592e138a125b179 P 63e3882aa0d1492c9f8aa623f2e4ce1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08cf15f24c4d4e638e5917ecd55eb90a, 63e3882aa0d1492c9f8aa623f2e4ce1d; no voters:
I20260620 18:40:34.817137 9454 raft_consensus.cc:2781] T 7274279c27e54f039592e138a125b179 P 63e3882aa0d1492c9f8aa623f2e4ce1d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:40:34.817631 9454 raft_consensus.cc:686] T 7274279c27e54f039592e138a125b179 P 63e3882aa0d1492c9f8aa623f2e4ce1d [term 1 LEADER]: Becoming Leader. State: Replica: 63e3882aa0d1492c9f8aa623f2e4ce1d, State: Running, Role: LEADER
I20260620 18:40:34.818087 9454 consensus_queue.cc:237] T 7274279c27e54f039592e138a125b179 P 63e3882aa0d1492c9f8aa623f2e4ce1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08cf15f24c4d4e638e5917ecd55eb90a" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 37183 } } peers { permanent_uuid: "63e3882aa0d1492c9f8aa623f2e4ce1d" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 35735 } } peers { permanent_uuid: "fc6d70339f694cfd8b046842726a0d66" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 42033 } }
I20260620 18:40:34.822793 8982 catalog_manager.cc:5495] T 7274279c27e54f039592e138a125b179 P 63e3882aa0d1492c9f8aa623f2e4ce1d reported cstate change: term changed from 0 to 1, leader changed from <none> to 63e3882aa0d1492c9f8aa623f2e4ce1d (127.22.37.1). New cstate: current_term: 1 leader_uuid: "63e3882aa0d1492c9f8aa623f2e4ce1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08cf15f24c4d4e638e5917ecd55eb90a" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 37183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63e3882aa0d1492c9f8aa623f2e4ce1d" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 35735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc6d70339f694cfd8b046842726a0d66" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 42033 } health_report { overall_health: UNKNOWN } } }
I20260620 18:40:34.843564 9444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:34.836906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60904 (local address 127.22.37.1:35735)
0620 18:40:34.837115 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:34.837120 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:34.837138 (+ 18us) server_negotiation.cc:407] Connection header received
0620 18:40:34.837360 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:34.837394 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:34.837509 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:34.837646 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:40:34.838002 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:34.838828 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:34.840924 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:34.841646 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:34.841796 (+ 150us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:34.842442 (+ 646us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:40:34.842934 (+ 492us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:40:34.843075 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:34.843261 (+ 186us) server_negotiation.cc:299] Negotiation successful
0620 18:40:34.843311 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 18:40:34.851851 9254 raft_consensus.cc:1261] T 7274279c27e54f039592e138a125b179 P fc6d70339f694cfd8b046842726a0d66 [term 1 FOLLOWER]: Refusing update from remote peer 63e3882aa0d1492c9f8aa623f2e4ce1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:40:34.851887 9384 raft_consensus.cc:1261] T 7274279c27e54f039592e138a125b179 P 08cf15f24c4d4e638e5917ecd55eb90a [term 1 FOLLOWER]: Refusing update from remote peer 63e3882aa0d1492c9f8aa623f2e4ce1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:40:34.852854 9458 consensus_queue.cc:1035] T 7274279c27e54f039592e138a125b179 P 63e3882aa0d1492c9f8aa623f2e4ce1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc6d70339f694cfd8b046842726a0d66" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 42033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:40:34.853233 9454 consensus_queue.cc:1035] T 7274279c27e54f039592e138a125b179 P 63e3882aa0d1492c9f8aa623f2e4ce1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "08cf15f24c4d4e638e5917ecd55eb90a" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 37183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 18:40:34.861235 9430 tablet.cc:2369] T 7274279c27e54f039592e138a125b179 P 08cf15f24c4d4e638e5917ecd55eb90a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:40:34.882221 8981 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42850:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:40:34.882566 8981 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42850:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:40:34.886611 8981 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 40903511f4984760a10503c1c746c588: Sending DeleteTablet for 3 replicas of tablet 7274279c27e54f039592e138a125b179
I20260620 18:40:34.887869 9364 tablet_service.cc:1514] Processing DeleteTablet for tablet 7274279c27e54f039592e138a125b179 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:40:34 UTC) from {username='slave'} at 127.0.0.1:45526
I20260620 18:40:34.887933 9234 tablet_service.cc:1514] Processing DeleteTablet for tablet 7274279c27e54f039592e138a125b179 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:40:34 UTC) from {username='slave'} at 127.0.0.1:42450
I20260620 18:40:34.888077 9103 tablet_service.cc:1514] Processing DeleteTablet for tablet 7274279c27e54f039592e138a125b179 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:40:34 UTC) from {username='slave'} at 127.0.0.1:60894
I20260620 18:40:34.890215 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 9041
I20260620 18:40:34.904206 9472 tablet_replica.cc:330] T 7274279c27e54f039592e138a125b179 P 63e3882aa0d1492c9f8aa623f2e4ce1d: stopping tablet replica
I20260620 18:40:34.904773 9472 raft_consensus.cc:2229] T 7274279c27e54f039592e138a125b179 P 63e3882aa0d1492c9f8aa623f2e4ce1d [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:40:34.905643 9472 raft_consensus.cc:2258] T 7274279c27e54f039592e138a125b179 P 63e3882aa0d1492c9f8aa623f2e4ce1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:40:34.908144 9472 ts_tablet_manager.cc:1905] T 7274279c27e54f039592e138a125b179 P 63e3882aa0d1492c9f8aa623f2e4ce1d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:40:34.917562 9472 ts_tablet_manager.cc:1918] T 7274279c27e54f039592e138a125b179 P 63e3882aa0d1492c9f8aa623f2e4ce1d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:40:34.917768 9472 log.cc:1196] T 7274279c27e54f039592e138a125b179 P 63e3882aa0d1492c9f8aa623f2e4ce1d: Deleting WAL directory at /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980527431848-22676-0/minicluster-data/ts-0/wal/wals/7274279c27e54f039592e138a125b179
I20260620 18:40:34.918335 9472 ts_tablet_manager.cc:1939] T 7274279c27e54f039592e138a125b179 P 63e3882aa0d1492c9f8aa623f2e4ce1d: Deleting consensus metadata
I20260620 18:40:34.920414 9471 tablet_replica.cc:330] T 7274279c27e54f039592e138a125b179 P fc6d70339f694cfd8b046842726a0d66: stopping tablet replica
I20260620 18:40:34.920861 9471 raft_consensus.cc:2229] T 7274279c27e54f039592e138a125b179 P fc6d70339f694cfd8b046842726a0d66 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:40:34.921427 9471 raft_consensus.cc:2258] T 7274279c27e54f039592e138a125b179 P fc6d70339f694cfd8b046842726a0d66 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:40:34.921452 8966 catalog_manager.cc:4841] TS 63e3882aa0d1492c9f8aa623f2e4ce1d (127.22.37.1:35735): tablet 7274279c27e54f039592e138a125b179 (table test-table [id=61d0da845ea54eb185b50657e20a7b07]) successfully deleted
I20260620 18:40:34.923053 9470 tablet_replica.cc:330] T 7274279c27e54f039592e138a125b179 P 08cf15f24c4d4e638e5917ecd55eb90a: stopping tablet replica
I20260620 18:40:34.923462 9470 raft_consensus.cc:2229] T 7274279c27e54f039592e138a125b179 P 08cf15f24c4d4e638e5917ecd55eb90a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:40:34.923547 9471 ts_tablet_manager.cc:1905] T 7274279c27e54f039592e138a125b179 P fc6d70339f694cfd8b046842726a0d66: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:40:34.923940 9470 raft_consensus.cc:2258] T 7274279c27e54f039592e138a125b179 P 08cf15f24c4d4e638e5917ecd55eb90a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:40:34.926446 9470 ts_tablet_manager.cc:1905] T 7274279c27e54f039592e138a125b179 P 08cf15f24c4d4e638e5917ecd55eb90a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:40:34.927814 9444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:34.891883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60908 (local address 127.22.37.1:35735)
0620 18:40:34.892116 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:34.892126 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:34.918419 (+ 26293us) server_negotiation.cc:407] Connection header received
0620 18:40:34.918797 (+ 378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:34.918803 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:34.918901 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:34.918993 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:40:34.919704 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:34.921056 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:34.923771 (+ 2715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:34.924445 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:34.924559 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:34.926125 (+ 1566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:40:34.926135 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:40:34.926153 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:40:34.926903 (+ 750us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:40:34.926969 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:40:34.926993 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:40:34.927065 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:34.927351 (+ 286us) server_negotiation.cc:299] Negotiation successful
0620 18:40:34.927471 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 18:40:34.929173 9163 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:40:34.931231 9471 ts_tablet_manager.cc:1918] T 7274279c27e54f039592e138a125b179 P fc6d70339f694cfd8b046842726a0d66: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:40:34.931419 9471 log.cc:1196] T 7274279c27e54f039592e138a125b179 P fc6d70339f694cfd8b046842726a0d66: Deleting WAL directory at /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980527431848-22676-0/minicluster-data/ts-1/wal/wals/7274279c27e54f039592e138a125b179
I20260620 18:40:34.932021 9471 ts_tablet_manager.cc:1939] T 7274279c27e54f039592e138a125b179 P fc6d70339f694cfd8b046842726a0d66: Deleting consensus metadata
I20260620 18:40:34.933760 9470 ts_tablet_manager.cc:1918] T 7274279c27e54f039592e138a125b179 P 08cf15f24c4d4e638e5917ecd55eb90a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:40:34.933900 9470 log.cc:1196] T 7274279c27e54f039592e138a125b179 P 08cf15f24c4d4e638e5917ecd55eb90a: Deleting WAL directory at /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980527431848-22676-0/minicluster-data/ts-2/wal/wals/7274279c27e54f039592e138a125b179
I20260620 18:40:34.933925 8966 catalog_manager.cc:4841] TS fc6d70339f694cfd8b046842726a0d66 (127.22.37.2:42033): tablet 7274279c27e54f039592e138a125b179 (table test-table [id=61d0da845ea54eb185b50657e20a7b07]) successfully deleted
I20260620 18:40:34.934368 9470 ts_tablet_manager.cc:1939] T 7274279c27e54f039592e138a125b179 P 08cf15f24c4d4e638e5917ecd55eb90a: Deleting consensus metadata
I20260620 18:40:34.936199 8969 catalog_manager.cc:4841] TS 08cf15f24c4d4e638e5917ecd55eb90a (127.22.37.3:37183): tablet 7274279c27e54f039592e138a125b179 (table test-table [id=61d0da845ea54eb185b50657e20a7b07]) successfully deleted
I20260620 18:40:35.027276 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 9041
I20260620 18:40:35.046375 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 9172
I20260620 18:40:35.053062 9445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:35.047197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42466 (local address 127.22.37.2:42033)
0620 18:40:35.047383 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:35.047388 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:35.047407 (+ 19us) server_negotiation.cc:407] Connection header received
0620 18:40:35.047549 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:35.047553 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:35.047646 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:35.047731 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:40:35.048081 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:35.048833 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:35.050324 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:35.050749 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:35.050843 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:35.051822 (+ 979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:40:35.051831 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:40:35.051847 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:40:35.052450 (+ 603us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:40:35.052499 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:40:35.052518 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:40:35.052612 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:35.052715 (+ 103us) server_negotiation.cc:299] Negotiation successful
0620 18:40:35.052810 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 18:40:35.053973 9294 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:40:35.147655 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 9172
I20260620 18:40:35.164819 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 9302
I20260620 18:40:35.177829 9442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:35.165532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45542 (local address 127.22.37.3:37183)
0620 18:40:35.165722 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:35.165730 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:35.165773 (+ 43us) server_negotiation.cc:407] Connection header received
0620 18:40:35.165989 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:35.165999 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:35.166098 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:35.166203 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:40:35.166626 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:35.167359 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:35.168607 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:35.168989 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:35.169205 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:35.176555 (+ 7350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:40:35.176564 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:40:35.176579 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:40:35.177130 (+ 551us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:40:35.177190 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:40:35.177207 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:40:35.177333 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:35.177449 (+ 116us) server_negotiation.cc:299] Negotiation successful
0620 18:40:35.177555 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 18:40:35.178438 9424 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:40:35.263569 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 9302
I20260620 18:40:35.280488 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 8951
I20260620 18:40:35.281670 9012 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:40:35.390189 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 8951
2026-06-20T18:40:35Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2761 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980527431848-22676-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:40:35 dist-test-slave-5lgp krb5kdc[9480](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:40:35 dist-test-slave-5lgp krb5kdc[9480](info): set up 2 sockets
Jun 20 18:40:35 dist-test-slave-5lgp krb5kdc[9480](info): commencing operation
krb5kdc: starting...
W20260620 18:40:36.481630 22676 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980527431848-22676-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980527431848-22676-0/krb5kdc/test-user.keytab.
Jun 20 18:40:36 dist-test-slave-5lgp krb5kdc[9480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:40:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:40:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.37.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.37.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.37.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab.
Entry for principal kudu/127.22.37.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab.
I20260620 18:40:36.729966 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980527431848-22676-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980527431848-22676-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980527431848-22676-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.62:43177
--webserver_interface=127.22.37.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab
--principal=kudu/127.22.37.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:39519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.37.62:43177
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:40:37.068774 9493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:40:37.069083 9493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:40:37.069242 9493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:40:37.076548 9493 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:40:37.076656 9493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:40:37.076740 9493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:40:37.076845 9493 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:40:37.076910 9493 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:40:37.086874 9493 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:39519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980527431848-22676-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980527431848-22676-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.37.62:43177
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab
--principal=kudu/127.22.37.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.62:43177
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.37.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980527431848-22676-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:40:37.088207 9493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:40:37.089533 9493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:40:37.101006 9499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:40:37.101095 9498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:40:37.101054 9501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:40:37.101840 9493 server_base.cc:1034] running on GCE node
I20260620 18:40:37.102602 9493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:40:37.103953 9493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:40:37.105201 9493 hybrid_clock.cc:648] HybridClock initialized: now 1781980837105159 us; error 54 us; skew 500 ppm
I20260620 18:40:37.105594 9493 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:40:37 dist-test-slave-5lgp krb5kdc[9480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980837, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:40:37.108589 9493 init.cc:377] Logged in from keytab as kudu/127.22.37.62@KRBTEST.COM (short username kudu)
I20260620 18:40:37.110051 9493 webserver.cc:460] Webserver started at http://127.22.37.62:46599/ using document root <none> and password file <none>
I20260620 18:40:37.110759 9493 fs_manager.cc:359] Metadata directory not provided
I20260620 18:40:37.110921 9493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:40:37.111227 9493 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:40:37.114202 9493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980527431848-22676-0/minicluster-data/master-0/data/instance:
uuid: "1cffb780af35489381efe222d20ff2b3"
format_stamp: "Formatted at 2026-06-20 18:40:37 on dist-test-slave-5lgp"
server_key: "815a34333000aef541a437428ce42466"
server_key_iv: "bddf312d76977463e5404dce6206cf06"
server_key_version: "encryptionkey@0"
I20260620 18:40:37.114938 9493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980527431848-22676-0/minicluster-data/master-0/wal/instance:
uuid: "1cffb780af35489381efe222d20ff2b3"
format_stamp: "Formatted at 2026-06-20 18:40:37 on dist-test-slave-5lgp"
server_key: "815a34333000aef541a437428ce42466"
server_key_iv: "bddf312d76977463e5404dce6206cf06"
server_key_version: "encryptionkey@0"
I20260620 18:40:37.120064 9493 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 18:40:37.123373 9508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:40:37.124832 9493 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:40:37.125016 9493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980527431848-22676-0/minicluster-data/master-0/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980527431848-22676-0/minicluster-data/master-0/wal
uuid: "1cffb780af35489381efe222d20ff2b3"
format_stamp: "Formatted at 2026-06-20 18:40:37 on dist-test-slave-5lgp"
server_key: "815a34333000aef541a437428ce42466"
server_key_iv: "bddf312d76977463e5404dce6206cf06"
server_key_version: "encryptionkey@0"
I20260620 18:40:37.125209 9493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980527431848-22676-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980527431848-22676-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980527431848-22676-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:40:37.144358 9493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:40:37.148653 9493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:40:37.148941 9493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:40:37.167333 9493 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.62:43177
I20260620 18:40:37.167374 9559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.62:43177 every 8 connection(s)
I20260620 18:40:37.168740 9493 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
I20260620 18:40:37.172495 9560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:40:37.174314 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 9493
I20260620 18:40:37.174558 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980527431848-22676-0/minicluster-data/master-0/wal/instance
I20260620 18:40:37.174995 22676 external_mini_cluster.cc:1428] Setting key ab701e191a2a84df6b8e1d68a6ce0e4c
I20260620 18:40:37.186972 9560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cffb780af35489381efe222d20ff2b3: Bootstrap starting.
Jun 20 18:40:37 dist-test-slave-5lgp krb5kdc[9480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM
I20260620 18:40:37.190176 9560 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1cffb780af35489381efe222d20ff2b3: Neither blocks nor log segments found. Creating new log.
I20260620 18:40:37.191468 9560 log.cc:824] T 00000000000000000000000000000000 P 1cffb780af35489381efe222d20ff2b3: Log is configured to *not* fsync() on all Append() calls
I20260620 18:40:37.194530 9560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cffb780af35489381efe222d20ff2b3: No bootstrap required, opened a new log
I20260620 18:40:37.198352 9563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:37.176945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59564 (local address 127.22.37.62:43177)
0620 18:40:37.177858 (+ 913us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:37.177880 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:37.177964 (+ 84us) server_negotiation.cc:407] Connection header received
0620 18:40:37.179720 (+ 1756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:37.179748 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:37.180254 (+ 506us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:37.180677 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:40:37.182120 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:37.184189 (+ 2069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:37.185606 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:37.186579 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:37.186759 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:37.190757 (+ 3998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:40:37.190767 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:40:37.190786 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:40:37.190833 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:40:37.194278 (+ 3445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:40:37.195051 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:40:37.195058 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:40:37.195064 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:40:37.195229 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:40:37.195635 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:40:37.195640 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:40:37.195642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:40:37.196157 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:40:37.196484 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:37.196918 (+ 434us) server_negotiation.cc:299] Negotiation successful
0620 18:40:37.197333 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":665,"thread_start_us":426,"threads_started":1}
I20260620 18:40:37.203408 9560 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1cffb780af35489381efe222d20ff2b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cffb780af35489381efe222d20ff2b3" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 43177 } }
I20260620 18:40:37.203893 9560 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1cffb780af35489381efe222d20ff2b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:40:37.204025 9560 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1cffb780af35489381efe222d20ff2b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cffb780af35489381efe222d20ff2b3, State: Initialized, Role: FOLLOWER
I20260620 18:40:37.204545 9560 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1cffb780af35489381efe222d20ff2b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cffb780af35489381efe222d20ff2b3" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 43177 } }
I20260620 18:40:37.204777 9560 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1cffb780af35489381efe222d20ff2b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:40:37.204902 9560 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1cffb780af35489381efe222d20ff2b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:40:37.205083 9560 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1cffb780af35489381efe222d20ff2b3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:40:37.207347 9560 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1cffb780af35489381efe222d20ff2b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cffb780af35489381efe222d20ff2b3" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 43177 } }
I20260620 18:40:37.207892 9560 leader_election.cc:302] T 00000000000000000000000000000000 P 1cffb780af35489381efe222d20ff2b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1cffb780af35489381efe222d20ff2b3; no voters:
I20260620 18:40:37.208482 9560 leader_election.cc:288] T 00000000000000000000000000000000 P 1cffb780af35489381efe222d20ff2b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:40:37.208640 9565 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1cffb780af35489381efe222d20ff2b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:40:37.209334 9565 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1cffb780af35489381efe222d20ff2b3 [term 1 LEADER]: Becoming Leader. State: Replica: 1cffb780af35489381efe222d20ff2b3, State: Running, Role: LEADER
I20260620 18:40:37.209936 9565 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1cffb780af35489381efe222d20ff2b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cffb780af35489381efe222d20ff2b3" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 43177 } }
I20260620 18:40:37.210806 9560 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1cffb780af35489381efe222d20ff2b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:40:37.213872 9567 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cffb780af35489381efe222d20ff2b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1cffb780af35489381efe222d20ff2b3. Latest consensus state: current_term: 1 leader_uuid: "1cffb780af35489381efe222d20ff2b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cffb780af35489381efe222d20ff2b3" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 43177 } } }
I20260620 18:40:37.214025 9566 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cffb780af35489381efe222d20ff2b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1cffb780af35489381efe222d20ff2b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cffb780af35489381efe222d20ff2b3" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 43177 } } }
I20260620 18:40:37.214320 9567 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cffb780af35489381efe222d20ff2b3 [sys.catalog]: This master's current role is: LEADER
I20260620 18:40:37.214371 9566 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cffb780af35489381efe222d20ff2b3 [sys.catalog]: This master's current role is: LEADER
I20260620 18:40:37.215643 9575 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:40:37.220440 9575 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:40:37.229178 9575 catalog_manager.cc:1347] Generated new cluster ID: bb953c94e09b4d90bcf2c0038b590a94
I20260620 18:40:37.229314 9575 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:40:37.241179 9575 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:40:37.242512 9575 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:40:37.252594 9575 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1cffb780af35489381efe222d20ff2b3: Generated new TSK 0
I20260620 18:40:37.253496 9575 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.37.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.37.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.37.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.1.keytab.
Entry for principal kudu/127.22.37.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.1.keytab.
I20260620 18:40:37.312685 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980527431848-22676-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980527431848-22676-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980527431848-22676-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980527431848-22676-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.37.1:0
--local_ip_for_outbound_sockets=127.22.37.1
--webserver_interface=127.22.37.1
--webserver_port=0
--tserver_master_addrs=127.22.37.62:43177
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.1.keytab
--principal=kudu/127.22.37.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:39519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:40:37.648048 9586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:40:37.648336 9586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:40:37.648432 9586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:40:37.655860 9586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:40:37.655977 9586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:40:37.656091 9586 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.37.1
I20260620 18:40:37.665939 9586 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:39519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980527431848-22676-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980527431848-22676-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.1.keytab
--principal=kudu/127.22.37.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.37.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980527431848-22676-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.37.1
--webserver_port=0
--tserver_master_addrs=127.22.37.62:43177
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.37.1
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980527431848-22676-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:40:37.667178 9586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:40:37.668447 9586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:40:37.681864 9591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:40:37.681885 9592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:40:37.681831 9594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:40:37.683120 9586 server_base.cc:1034] running on GCE node
I20260620 18:40:37.683748 9586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:40:37.684720 9586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:40:37.685945 9586 hybrid_clock.cc:648] HybridClock initialized: now 1781980837685875 us; error 63 us; skew 500 ppm
I20260620 18:40:37.686269 9586 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:40:37 dist-test-slave-5lgp krb5kdc[9480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980837, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:40:37.689730 9586 init.cc:377] Logged in from keytab as kudu/127.22.37.1@KRBTEST.COM (short username kudu)
I20260620 18:40:37.691272 9586 webserver.cc:460] Webserver started at http://127.22.37.1:46651/ using document root <none> and password file <none>
I20260620 18:40:37.691975 9586 fs_manager.cc:359] Metadata directory not provided
I20260620 18:40:37.692122 9586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:40:37.692453 9586 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:40:37.695564 9586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980527431848-22676-0/minicluster-data/ts-0/data/instance:
uuid: "335a53e810cc4fe4a7bc614a357e7db3"
format_stamp: "Formatted at 2026-06-20 18:40:37 on dist-test-slave-5lgp"
server_key: "a2dc1668d9ba47b5907ec0ae6175a87f"
server_key_iv: "2d90390e42f5d0ed0bedc5f0ba7b2503"
server_key_version: "encryptionkey@0"
I20260620 18:40:37.696522 9586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980527431848-22676-0/minicluster-data/ts-0/wal/instance:
uuid: "335a53e810cc4fe4a7bc614a357e7db3"
format_stamp: "Formatted at 2026-06-20 18:40:37 on dist-test-slave-5lgp"
server_key: "a2dc1668d9ba47b5907ec0ae6175a87f"
server_key_iv: "2d90390e42f5d0ed0bedc5f0ba7b2503"
server_key_version: "encryptionkey@0"
I20260620 18:40:37.701813 9586 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 18:40:37.705220 9601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:40:37.706647 9586 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:40:37.706830 9586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980527431848-22676-0/minicluster-data/ts-0/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980527431848-22676-0/minicluster-data/ts-0/wal
uuid: "335a53e810cc4fe4a7bc614a357e7db3"
format_stamp: "Formatted at 2026-06-20 18:40:37 on dist-test-slave-5lgp"
server_key: "a2dc1668d9ba47b5907ec0ae6175a87f"
server_key_iv: "2d90390e42f5d0ed0bedc5f0ba7b2503"
server_key_version: "encryptionkey@0"
I20260620 18:40:37.707065 9586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980527431848-22676-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980527431848-22676-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980527431848-22676-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:40:37.719192 9586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:40:37.723093 9586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:40:37.723371 9586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:40:37.724272 9586 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:40:37.725668 9586 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:40:37.725755 9586 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:40:37.725870 9586 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:40:37.725940 9586 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:40:37.759029 9586 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.1:42841
I20260620 18:40:37.759128 9713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.1:42841 every 8 connection(s)
I20260620 18:40:37.760329 9586 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980527431848-22676-0/minicluster-data/ts-0/data/info.pb
I20260620 18:40:37.766992 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 9586
I20260620 18:40:37.767184 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980527431848-22676-0/minicluster-data/ts-0/wal/instance
I20260620 18:40:37.767640 22676 external_mini_cluster.cc:1428] Setting key 88f63c42f3906d9fba54ea844b5f8255
Jun 20 18:40:37 dist-test-slave-5lgp krb5kdc[9480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980837, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.1@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM
I20260620 18:40:37.782467 9716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:37.762934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.1:39937 (local address 127.22.37.62:43177)
0620 18:40:37.763388 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:37.763393 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:37.764270 (+ 877us) server_negotiation.cc:407] Connection header received
0620 18:40:37.766170 (+ 1900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:37.766175 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:37.766265 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:37.766383 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:40:37.768059 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:37.769138 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:37.771631 (+ 2493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:37.772032 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:37.772145 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:37.776375 (+ 4230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:40:37.776380 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:40:37.776384 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:40:37.776417 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:40:37.779179 (+ 2762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:40:37.779967 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:40:37.779971 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:40:37.779974 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:40:37.780105 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:40:37.780563 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:40:37.780570 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:40:37.780578 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:40:37.780860 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:40:37.781034 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:37.781837 (+ 803us) server_negotiation.cc:299] Negotiation successful
0620 18:40:37.782093 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":246,"threads_started":1}
I20260620 18:40:37.783457 9717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:37.763352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:43177 (local address 127.22.37.1:39937)
0620 18:40:37.764086 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:37.764131 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 18:40:37.765901 (+ 1770us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:40:37.766637 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:40:37.766645 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:40:37.767247 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:40:37.767885 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:37.767903 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:37.769350 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:37.769356 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:37.771460 (+ 2104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:37.771484 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:37.772278 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:37.772287 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:37.772610 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:37.773420 (+ 810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:40:37.773457 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:40:37.776127 (+ 2670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:40:37.779342 (+ 3215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:40:37.779352 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:40:37.779370 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:40:37.779821 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:40:37.780225 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:40:37.780231 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:40:37.780235 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:40:37.780431 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:40:37.780993 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:40:37.781002 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:40:37.781354 (+ 352us) client_negotiation.cc:769] Sending connection context
0620 18:40:37.781723 (+ 369us) client_negotiation.cc:240] Negotiation successful
0620 18:40:37.782278 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":294,"threads_started":1}
I20260620 18:40:37.785277 9714 heartbeater.cc:347] Connected to a master server at 127.22.37.62:43177
I20260620 18:40:37.785679 9714 heartbeater.cc:464] Registering TS with master...
I20260620 18:40:37.786574 9714 heartbeater.cc:511] Master 127.22.37.62:43177 requested a full tablet report, sending...
I20260620 18:40:37.789070 9525 ts_manager.cc:194] Registered new tserver with Master: 335a53e810cc4fe4a7bc614a357e7db3 (127.22.37.1:42841)
I20260620 18:40:37.791348 9525 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.37.1@KRBTEST.COM'} at 127.22.37.1:39937
WARNING: no policy specified for kudu/127.22.37.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.37.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.37.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.2.keytab.
Entry for principal kudu/127.22.37.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.2.keytab.
I20260620 18:40:37.828440 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980527431848-22676-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980527431848-22676-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980527431848-22676-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980527431848-22676-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.37.2:0
--local_ip_for_outbound_sockets=127.22.37.2
--webserver_interface=127.22.37.2
--webserver_port=0
--tserver_master_addrs=127.22.37.62:43177
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.2.keytab
--principal=kudu/127.22.37.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:39519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:40:38.159430 9720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:40:38.159768 9720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:40:38.159936 9720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:40:38.167213 9720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:40:38.167361 9720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:40:38.167560 9720 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.37.2
I20260620 18:40:38.177266 9720 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:39519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980527431848-22676-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980527431848-22676-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.2.keytab
--principal=kudu/127.22.37.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.37.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980527431848-22676-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.37.2
--webserver_port=0
--tserver_master_addrs=127.22.37.62:43177
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.37.2
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980527431848-22676-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:40:38.178608 9720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:40:38.179955 9720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:40:38.193346 9728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:40:38.193300 9725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:40:38.193328 9726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:40:38.194260 9720 server_base.cc:1034] running on GCE node
I20260620 18:40:38.194864 9720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:40:38.195801 9720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:40:38.197024 9720 hybrid_clock.cc:648] HybridClock initialized: now 1781980838196981 us; error 59 us; skew 500 ppm
I20260620 18:40:38.197408 9720 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:40:38 dist-test-slave-5lgp krb5kdc[9480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980838, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:40:38.200594 9720 init.cc:377] Logged in from keytab as kudu/127.22.37.2@KRBTEST.COM (short username kudu)
I20260620 18:40:38.202256 9720 webserver.cc:460] Webserver started at http://127.22.37.2:33089/ using document root <none> and password file <none>
I20260620 18:40:38.202970 9720 fs_manager.cc:359] Metadata directory not provided
I20260620 18:40:38.203132 9720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:40:38.203457 9720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:40:38.206709 9720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980527431848-22676-0/minicluster-data/ts-1/data/instance:
uuid: "e7fd899ef9a5456bba621fe7e5f5a1c8"
format_stamp: "Formatted at 2026-06-20 18:40:38 on dist-test-slave-5lgp"
server_key: "13f1dae4d7c61118d5727562f4470ad0"
server_key_iv: "e4f401db4dc3b9d0473e98e3db40bfd2"
server_key_version: "encryptionkey@0"
I20260620 18:40:38.207479 9720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980527431848-22676-0/minicluster-data/ts-1/wal/instance:
uuid: "e7fd899ef9a5456bba621fe7e5f5a1c8"
format_stamp: "Formatted at 2026-06-20 18:40:38 on dist-test-slave-5lgp"
server_key: "13f1dae4d7c61118d5727562f4470ad0"
server_key_iv: "e4f401db4dc3b9d0473e98e3db40bfd2"
server_key_version: "encryptionkey@0"
I20260620 18:40:38.213060 9720 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260620 18:40:38.216715 9735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:40:38.218163 9720 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:40:38.218339 9720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980527431848-22676-0/minicluster-data/ts-1/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980527431848-22676-0/minicluster-data/ts-1/wal
uuid: "e7fd899ef9a5456bba621fe7e5f5a1c8"
format_stamp: "Formatted at 2026-06-20 18:40:38 on dist-test-slave-5lgp"
server_key: "13f1dae4d7c61118d5727562f4470ad0"
server_key_iv: "e4f401db4dc3b9d0473e98e3db40bfd2"
server_key_version: "encryptionkey@0"
I20260620 18:40:38.218567 9720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980527431848-22676-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980527431848-22676-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980527431848-22676-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:40:38.272455 9720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:40:38.277024 9720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:40:38.277331 9720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:40:38.278359 9720 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:40:38.279712 9720 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:40:38.279816 9720 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:40:38.279966 9720 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:40:38.280040 9720 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:40:38.312511 9720 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.2:43331
I20260620 18:40:38.312565 9847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.2:43331 every 8 connection(s)
I20260620 18:40:38.314049 9720 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980527431848-22676-0/minicluster-data/ts-1/data/info.pb
I20260620 18:40:38.315732 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 9720
I20260620 18:40:38.315927 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980527431848-22676-0/minicluster-data/ts-1/wal/instance
I20260620 18:40:38.316370 22676 external_mini_cluster.cc:1428] Setting key 39dbf0cefdec3b32ff585f48de6d20fa
Jun 20 18:40:38 dist-test-slave-5lgp krb5kdc[9480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980838, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.2@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM
I20260620 18:40:38.336783 9850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:38.316854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.2:52007 (local address 127.22.37.62:43177)
0620 18:40:38.317316 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:38.317320 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:38.318540 (+ 1220us) server_negotiation.cc:407] Connection header received
0620 18:40:38.320538 (+ 1998us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:38.320553 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:38.320656 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:38.320783 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:40:38.322431 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:38.323625 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:38.325115 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:38.325902 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:38.326046 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:38.330089 (+ 4043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:40:38.330098 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:40:38.330107 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:40:38.330153 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:40:38.333775 (+ 3622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:40:38.334469 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:40:38.334475 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:40:38.334478 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:40:38.334662 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:40:38.335087 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:40:38.335093 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:40:38.335096 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:40:38.335366 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:40:38.335525 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:38.336199 (+ 674us) server_negotiation.cc:299] Negotiation successful
0620 18:40:38.336419 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":228,"threads_started":1}
I20260620 18:40:38.337787 9851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:38.317580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:43177 (local address 127.22.37.2:52007)
0620 18:40:38.318385 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:38.318428 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 18:40:38.320268 (+ 1840us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:40:38.320979 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:40:38.320987 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:40:38.321587 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:40:38.322261 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:38.322279 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:38.323758 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:38.323762 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:38.324967 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:38.324977 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:38.326098 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:38.326102 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:38.326300 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:38.327049 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:40:38.327083 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:40:38.329858 (+ 2775us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:40:38.333916 (+ 4058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:40:38.333922 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:40:38.333935 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:40:38.334335 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:40:38.334772 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:40:38.334778 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:40:38.334781 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:40:38.334958 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:40:38.335499 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:40:38.335506 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:40:38.335827 (+ 321us) client_negotiation.cc:769] Sending connection context
0620 18:40:38.336093 (+ 266us) client_negotiation.cc:240] Negotiation successful
0620 18:40:38.336620 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":354,"threads_started":1}
I20260620 18:40:38.339365 9848 heartbeater.cc:347] Connected to a master server at 127.22.37.62:43177
I20260620 18:40:38.339669 9848 heartbeater.cc:464] Registering TS with master...
I20260620 18:40:38.340373 9848 heartbeater.cc:511] Master 127.22.37.62:43177 requested a full tablet report, sending...
I20260620 18:40:38.341986 9525 ts_manager.cc:194] Registered new tserver with Master: e7fd899ef9a5456bba621fe7e5f5a1c8 (127.22.37.2:43331)
WARNING: no policy specified for kudu/127.22.37.3@KRBTEST.COM; defaulting to no policy
I20260620 18:40:38.343221 9525 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.37.2@KRBTEST.COM'} at 127.22.37.2:52007
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.37.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.37.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.3.keytab.
Entry for principal kudu/127.22.37.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.3.keytab.
I20260620 18:40:38.372376 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980527431848-22676-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980527431848-22676-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980527431848-22676-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980527431848-22676-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.37.3:0
--local_ip_for_outbound_sockets=127.22.37.3
--webserver_interface=127.22.37.3
--webserver_port=0
--tserver_master_addrs=127.22.37.62:43177
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.3.keytab
--principal=kudu/127.22.37.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:39519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:40:38.691236 9854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:40:38.691509 9854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:40:38.691660 9854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:40:38.698947 9854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:40:38.699050 9854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:40:38.699225 9854 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.37.3
I20260620 18:40:38.708702 9854 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:39519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980527431848-22676-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980527431848-22676-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.3.keytab
--principal=kudu/127.22.37.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.37.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980527431848-22676-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.37.3
--webserver_port=0
--tserver_master_addrs=127.22.37.62:43177
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.37.3
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980527431848-22676-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:40:38.709955 9854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:40:38.711365 9854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:40:38.724506 9862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:40:38.724545 9859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:40:38.724578 9860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:40:38.725806 9854 server_base.cc:1034] running on GCE node
I20260620 18:40:38.726380 9854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:40:38.727335 9854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:40:38.728546 9854 hybrid_clock.cc:648] HybridClock initialized: now 1781980838728502 us; error 41 us; skew 500 ppm
I20260620 18:40:38.728884 9854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:40:38 dist-test-slave-5lgp krb5kdc[9480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980838, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:40:38.732036 9854 init.cc:377] Logged in from keytab as kudu/127.22.37.3@KRBTEST.COM (short username kudu)
I20260620 18:40:38.733494 9854 webserver.cc:460] Webserver started at http://127.22.37.3:33301/ using document root <none> and password file <none>
I20260620 18:40:38.734189 9854 fs_manager.cc:359] Metadata directory not provided
I20260620 18:40:38.734337 9854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:40:38.734661 9854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:40:38.737802 9854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980527431848-22676-0/minicluster-data/ts-2/data/instance:
uuid: "7c69cc4810c74ffe8d9dfc25964557b5"
format_stamp: "Formatted at 2026-06-20 18:40:38 on dist-test-slave-5lgp"
server_key: "0a4557f75b127083b6813db55da368aa"
server_key_iv: "d68f89a15bf494b93adf1b4ab64278e9"
server_key_version: "encryptionkey@0"
I20260620 18:40:38.738566 9854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980527431848-22676-0/minicluster-data/ts-2/wal/instance:
uuid: "7c69cc4810c74ffe8d9dfc25964557b5"
format_stamp: "Formatted at 2026-06-20 18:40:38 on dist-test-slave-5lgp"
server_key: "0a4557f75b127083b6813db55da368aa"
server_key_iv: "d68f89a15bf494b93adf1b4ab64278e9"
server_key_version: "encryptionkey@0"
I20260620 18:40:38.744400 9854 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 18:40:38.747824 9869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:40:38.749109 9854 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:40:38.749276 9854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980527431848-22676-0/minicluster-data/ts-2/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980527431848-22676-0/minicluster-data/ts-2/wal
uuid: "7c69cc4810c74ffe8d9dfc25964557b5"
format_stamp: "Formatted at 2026-06-20 18:40:38 on dist-test-slave-5lgp"
server_key: "0a4557f75b127083b6813db55da368aa"
server_key_iv: "d68f89a15bf494b93adf1b4ab64278e9"
server_key_version: "encryptionkey@0"
I20260620 18:40:38.749536 9854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980527431848-22676-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980527431848-22676-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980527431848-22676-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:40:38.779678 9854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:40:38.783797 9854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:40:38.784111 9854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:40:38.785136 9854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:40:38.786558 9854 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:40:38.786674 9854 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:40:38.786834 9854 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:40:38.786908 9854 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:40:38.795436 9714 heartbeater.cc:503] Master 127.22.37.62:43177 was elected leader, sending a full tablet report...
I20260620 18:40:38.820950 9854 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.3:42891
I20260620 18:40:38.821002 9981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.3:42891 every 8 connection(s)
I20260620 18:40:38.822441 9854 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980527431848-22676-0/minicluster-data/ts-2/data/info.pb
I20260620 18:40:38.825436 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 9854
I20260620 18:40:38.825623 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980527431848-22676-0/minicluster-data/ts-2/wal/instance
I20260620 18:40:38.826118 22676 external_mini_cluster.cc:1428] Setting key 206f7ddd71385aa99cab179f77894280
Jun 20 18:40:38 dist-test-slave-5lgp krb5kdc[9480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980838, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.3@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM
I20260620 18:40:38.842962 9850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:38.825256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.3:57235 (local address 127.22.37.62:43177)
0620 18:40:38.825519 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:38.825526 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:38.826833 (+ 1307us) server_negotiation.cc:407] Connection header received
0620 18:40:38.828868 (+ 2035us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:38.828873 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:38.828979 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:38.829089 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:40:38.830725 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:38.831471 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:38.833009 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:38.833469 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:38.833606 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:38.837156 (+ 3550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:40:38.837161 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:40:38.837164 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:40:38.837197 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:40:38.839869 (+ 2672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:40:38.840618 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:40:38.840624 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:40:38.840627 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:40:38.840774 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:40:38.841194 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:40:38.841201 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:40:38.841204 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:40:38.841553 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:40:38.841722 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:38.842394 (+ 672us) server_negotiation.cc:299] Negotiation successful
0620 18:40:38.842621 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 18:40:38.844048 9984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:38.825835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:43177 (local address 127.22.37.3:57235)
0620 18:40:38.826670 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:38.826718 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 18:40:38.828593 (+ 1875us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:40:38.829338 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:40:38.829347 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:40:38.829986 (+ 639us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:40:38.830560 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:38.830576 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:38.831628 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:38.831634 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:38.832862 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:38.832874 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:38.833702 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:38.833709 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:38.833944 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:38.834648 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:40:38.834670 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:40:38.836961 (+ 2291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:40:38.840028 (+ 3067us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:40:38.840036 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:40:38.840054 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:40:38.840485 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:40:38.840887 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:40:38.840892 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:40:38.840896 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:40:38.841083 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:40:38.841679 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:40:38.841688 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:40:38.842034 (+ 346us) client_negotiation.cc:769] Sending connection context
0620 18:40:38.842308 (+ 274us) client_negotiation.cc:240] Negotiation successful
0620 18:40:38.842856 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":353,"threads_started":1}
I20260620 18:40:38.845463 9982 heartbeater.cc:347] Connected to a master server at 127.22.37.62:43177
I20260620 18:40:38.845774 9982 heartbeater.cc:464] Registering TS with master...
I20260620 18:40:38.846462 9982 heartbeater.cc:511] Master 127.22.37.62:43177 requested a full tablet report, sending...
I20260620 18:40:38.847896 9525 ts_manager.cc:194] Registered new tserver with Master: 7c69cc4810c74ffe8d9dfc25964557b5 (127.22.37.3:42891)
I20260620 18:40:38.848716 9525 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.37.3@KRBTEST.COM'} at 127.22.37.3:57235
I20260620 18:40:38.853971 22676 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:40:38.871420 9850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:38.858752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39644 (local address 127.22.37.62:43177)
0620 18:40:38.858931 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:38.858935 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:38.859382 (+ 447us) server_negotiation.cc:407] Connection header received
0620 18:40:38.859523 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:38.859526 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:38.859600 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:38.859690 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:40:38.861130 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:38.861973 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:38.863313 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:38.863766 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:38.863879 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:38.866318 (+ 2439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:40:38.866323 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:40:38.866326 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:40:38.866358 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:40:38.868979 (+ 2621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:40:38.869658 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:40:38.869662 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:40:38.869664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:40:38.869781 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:40:38.870201 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:40:38.870205 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:40:38.870206 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:40:38.870507 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:40:38.870697 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:38.870981 (+ 284us) server_negotiation.cc:299] Negotiation successful
0620 18:40:38.871171 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 18:40:38.875402 9525 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39644:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:40:38.882324 9525 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:40:38.922633 9997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:38.894626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.2:43331 (local address 127.0.0.1:47996)
0620 18:40:38.905005 (+ 10379us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:38.905085 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0620 18:40:38.905306 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:40:38.915573 (+ 10267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:40:38.915584 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:40:38.915623 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:40:38.916059 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:38.916069 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:38.918067 (+ 1998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:38.918074 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:38.920096 (+ 2022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:38.920107 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:38.921467 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:38.921474 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:38.921757 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:38.921855 (+ 98us) client_negotiation.cc:769] Sending connection context
0620 18:40:38.922004 (+ 149us) client_negotiation.cc:240] Negotiation successful
0620 18:40:38.922195 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10038,"mutex_wait_us":62,"thread_start_us":20543,"threads_started":1}
I20260620 18:40:38.923113 9995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:38.894879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.1:42841 (local address 127.0.0.1:45444)
0620 18:40:38.914528 (+ 19649us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:38.914550 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 18:40:38.914691 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:40:38.915799 (+ 1108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:40:38.915804 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:40:38.915832 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:40:38.916316 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:38.916330 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:38.918100 (+ 1770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:38.918108 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:38.919921 (+ 1813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:38.919935 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:38.921971 (+ 2036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:38.921977 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:38.922142 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:38.922165 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 18:40:38.922265 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 18:40:38.922377 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19497,"mutex_wait_us":26,"thread_start_us":19368,"threads_started":1}
I20260620 18:40:38.923243 9999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:38.896011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45444 (local address 127.22.37.1:42841)
0620 18:40:38.908138 (+ 12127us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:38.908151 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:38.914675 (+ 6524us) server_negotiation.cc:407] Connection header received
0620 18:40:38.915148 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:38.915160 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:38.915431 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:38.915632 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:40:38.916496 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:38.917996 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:38.920113 (+ 2117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:38.921612 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:38.921733 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:38.921984 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:38.922312 (+ 328us) server_negotiation.cc:299] Negotiation successful
0620 18:40:38.922468 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11914,"thread_start_us":11755,"threads_started":1}
I20260620 18:40:38.923399 9996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:38.894842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47996 (local address 127.22.37.2:43331)
0620 18:40:38.914839 (+ 19997us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:38.914847 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:38.914873 (+ 26us) server_negotiation.cc:407] Connection header received
0620 18:40:38.914988 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:38.914999 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:38.915327 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:38.915540 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:40:38.916251 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:38.917894 (+ 1643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:38.920245 (+ 2351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:38.921228 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:38.921360 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:38.921618 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:38.922855 (+ 1237us) server_negotiation.cc:299] Negotiation successful
0620 18:40:38.923046 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19819,"thread_start_us":19949,"threads_started":1}
I20260620 18:40:38.924584 9994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:38.895437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.3:42891 (local address 127.0.0.1:53196)
0620 18:40:38.915468 (+ 20031us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:38.915486 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:40:38.915593 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:40:38.916258 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:40:38.916264 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:40:38.916294 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:40:38.916629 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:38.916637 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:38.919523 (+ 2886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:38.919528 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:38.921724 (+ 2196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:38.921737 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:38.923713 (+ 1976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:38.923720 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:38.923899 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:38.923929 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 18:40:38.924077 (+ 148us) client_negotiation.cc:240] Negotiation successful
0620 18:40:38.924193 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19900,"mutex_wait_us":3,"thread_start_us":9054,"threads_started":1}
I20260620 18:40:38.924623 9998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:38.895763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53196 (local address 127.22.37.3:42891)
0620 18:40:38.909598 (+ 13835us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:38.909610 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:38.915723 (+ 6113us) server_negotiation.cc:407] Connection header received
0620 18:40:38.915823 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:38.915829 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:38.916044 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:38.916768 (+ 724us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:40:38.916962 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:38.919366 (+ 2404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:38.921894 (+ 2528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:38.923523 (+ 1629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:38.923650 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:38.923864 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:38.924128 (+ 264us) server_negotiation.cc:299] Negotiation successful
0620 18:40:38.924273 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13610,"thread_start_us":13839,"threads_started":1}
I20260620 18:40:38.927065 9783 tablet_service.cc:1467] Processing CreateTablet for tablet 4cf1432d2825431093a508043f4a0185 (DEFAULT_TABLE table=test-table [id=9c40bfa28ac6443289a4e9186cbca458]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:40:38.928309 9783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cf1432d2825431093a508043f4a0185. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:40:38.928251 9917 tablet_service.cc:1467] Processing CreateTablet for tablet 4cf1432d2825431093a508043f4a0185 (DEFAULT_TABLE table=test-table [id=9c40bfa28ac6443289a4e9186cbca458]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:40:38.928251 9649 tablet_service.cc:1467] Processing CreateTablet for tablet 4cf1432d2825431093a508043f4a0185 (DEFAULT_TABLE table=test-table [id=9c40bfa28ac6443289a4e9186cbca458]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:40:38.929675 9649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cf1432d2825431093a508043f4a0185. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:40:38.929677 9917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cf1432d2825431093a508043f4a0185. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:40:38.939402 10000 tablet_bootstrap.cc:492] T 4cf1432d2825431093a508043f4a0185 P e7fd899ef9a5456bba621fe7e5f5a1c8: Bootstrap starting.
I20260620 18:40:38.942529 10000 tablet_bootstrap.cc:654] T 4cf1432d2825431093a508043f4a0185 P e7fd899ef9a5456bba621fe7e5f5a1c8: Neither blocks nor log segments found. Creating new log.
I20260620 18:40:38.943830 10000 log.cc:824] T 4cf1432d2825431093a508043f4a0185 P e7fd899ef9a5456bba621fe7e5f5a1c8: Log is configured to *not* fsync() on all Append() calls
I20260620 18:40:38.949914 10002 tablet_bootstrap.cc:492] T 4cf1432d2825431093a508043f4a0185 P 7c69cc4810c74ffe8d9dfc25964557b5: Bootstrap starting.
I20260620 18:40:38.949914 10001 tablet_bootstrap.cc:492] T 4cf1432d2825431093a508043f4a0185 P 335a53e810cc4fe4a7bc614a357e7db3: Bootstrap starting.
I20260620 18:40:38.952199 10000 tablet_bootstrap.cc:492] T 4cf1432d2825431093a508043f4a0185 P e7fd899ef9a5456bba621fe7e5f5a1c8: No bootstrap required, opened a new log
I20260620 18:40:38.952473 10000 ts_tablet_manager.cc:1397] T 4cf1432d2825431093a508043f4a0185 P e7fd899ef9a5456bba621fe7e5f5a1c8: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.004s
I20260620 18:40:38.953817 10001 tablet_bootstrap.cc:654] T 4cf1432d2825431093a508043f4a0185 P 335a53e810cc4fe4a7bc614a357e7db3: Neither blocks nor log segments found. Creating new log.
I20260620 18:40:38.953824 10002 tablet_bootstrap.cc:654] T 4cf1432d2825431093a508043f4a0185 P 7c69cc4810c74ffe8d9dfc25964557b5: Neither blocks nor log segments found. Creating new log.
I20260620 18:40:38.955047 10002 log.cc:824] T 4cf1432d2825431093a508043f4a0185 P 7c69cc4810c74ffe8d9dfc25964557b5: Log is configured to *not* fsync() on all Append() calls
I20260620 18:40:38.955047 10001 log.cc:824] T 4cf1432d2825431093a508043f4a0185 P 335a53e810cc4fe4a7bc614a357e7db3: Log is configured to *not* fsync() on all Append() calls
I20260620 18:40:38.958583 10001 tablet_bootstrap.cc:492] T 4cf1432d2825431093a508043f4a0185 P 335a53e810cc4fe4a7bc614a357e7db3: No bootstrap required, opened a new log
I20260620 18:40:38.958859 10001 ts_tablet_manager.cc:1397] T 4cf1432d2825431093a508043f4a0185 P 335a53e810cc4fe4a7bc614a357e7db3: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260620 18:40:38.959019 10002 tablet_bootstrap.cc:492] T 4cf1432d2825431093a508043f4a0185 P 7c69cc4810c74ffe8d9dfc25964557b5: No bootstrap required, opened a new log
I20260620 18:40:38.959268 10002 ts_tablet_manager.cc:1397] T 4cf1432d2825431093a508043f4a0185 P 7c69cc4810c74ffe8d9dfc25964557b5: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260620 18:40:38.962718 10000 raft_consensus.cc:348] T 4cf1432d2825431093a508043f4a0185 P e7fd899ef9a5456bba621fe7e5f5a1c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7fd899ef9a5456bba621fe7e5f5a1c8" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 43331 } } peers { permanent_uuid: "335a53e810cc4fe4a7bc614a357e7db3" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 42841 } } peers { permanent_uuid: "7c69cc4810c74ffe8d9dfc25964557b5" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 42891 } }
I20260620 18:40:38.963201 10000 raft_consensus.cc:374] T 4cf1432d2825431093a508043f4a0185 P e7fd899ef9a5456bba621fe7e5f5a1c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:40:38.963299 10000 raft_consensus.cc:729] T 4cf1432d2825431093a508043f4a0185 P e7fd899ef9a5456bba621fe7e5f5a1c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7fd899ef9a5456bba621fe7e5f5a1c8, State: Initialized, Role: FOLLOWER
I20260620 18:40:38.963901 10000 consensus_queue.cc:260] T 4cf1432d2825431093a508043f4a0185 P e7fd899ef9a5456bba621fe7e5f5a1c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7fd899ef9a5456bba621fe7e5f5a1c8" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 43331 } } peers { permanent_uuid: "335a53e810cc4fe4a7bc614a357e7db3" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 42841 } } peers { permanent_uuid: "7c69cc4810c74ffe8d9dfc25964557b5" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 42891 } }
I20260620 18:40:38.964994 9848 heartbeater.cc:503] Master 127.22.37.62:43177 was elected leader, sending a full tablet report...
I20260620 18:40:38.965695 10000 ts_tablet_manager.cc:1428] T 4cf1432d2825431093a508043f4a0185 P e7fd899ef9a5456bba621fe7e5f5a1c8: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260620 18:40:38.967656 10001 raft_consensus.cc:348] T 4cf1432d2825431093a508043f4a0185 P 335a53e810cc4fe4a7bc614a357e7db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7fd899ef9a5456bba621fe7e5f5a1c8" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 43331 } } peers { permanent_uuid: "335a53e810cc4fe4a7bc614a357e7db3" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 42841 } } peers { permanent_uuid: "7c69cc4810c74ffe8d9dfc25964557b5" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 42891 } }
I20260620 18:40:38.968034 10001 raft_consensus.cc:374] T 4cf1432d2825431093a508043f4a0185 P 335a53e810cc4fe4a7bc614a357e7db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:40:38.968186 10001 raft_consensus.cc:729] T 4cf1432d2825431093a508043f4a0185 P 335a53e810cc4fe4a7bc614a357e7db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 335a53e810cc4fe4a7bc614a357e7db3, State: Initialized, Role: FOLLOWER
I20260620 18:40:38.968773 10001 consensus_queue.cc:260] T 4cf1432d2825431093a508043f4a0185 P 335a53e810cc4fe4a7bc614a357e7db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7fd899ef9a5456bba621fe7e5f5a1c8" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 43331 } } peers { permanent_uuid: "335a53e810cc4fe4a7bc614a357e7db3" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 42841 } } peers { permanent_uuid: "7c69cc4810c74ffe8d9dfc25964557b5" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 42891 } }
I20260620 18:40:38.969419 10002 raft_consensus.cc:348] T 4cf1432d2825431093a508043f4a0185 P 7c69cc4810c74ffe8d9dfc25964557b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7fd899ef9a5456bba621fe7e5f5a1c8" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 43331 } } peers { permanent_uuid: "335a53e810cc4fe4a7bc614a357e7db3" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 42841 } } peers { permanent_uuid: "7c69cc4810c74ffe8d9dfc25964557b5" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 42891 } }
I20260620 18:40:38.969833 10002 raft_consensus.cc:374] T 4cf1432d2825431093a508043f4a0185 P 7c69cc4810c74ffe8d9dfc25964557b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:40:38.969929 10002 raft_consensus.cc:729] T 4cf1432d2825431093a508043f4a0185 P 7c69cc4810c74ffe8d9dfc25964557b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c69cc4810c74ffe8d9dfc25964557b5, State: Initialized, Role: FOLLOWER
I20260620 18:40:38.970561 10001 ts_tablet_manager.cc:1428] T 4cf1432d2825431093a508043f4a0185 P 335a53e810cc4fe4a7bc614a357e7db3: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260620 18:40:38.970577 10002 consensus_queue.cc:260] T 4cf1432d2825431093a508043f4a0185 P 7c69cc4810c74ffe8d9dfc25964557b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7fd899ef9a5456bba621fe7e5f5a1c8" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 43331 } } peers { permanent_uuid: "335a53e810cc4fe4a7bc614a357e7db3" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 42841 } } peers { permanent_uuid: "7c69cc4810c74ffe8d9dfc25964557b5" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 42891 } }
I20260620 18:40:38.971570 9982 heartbeater.cc:503] Master 127.22.37.62:43177 was elected leader, sending a full tablet report...
I20260620 18:40:38.972110 10002 ts_tablet_manager.cc:1428] T 4cf1432d2825431093a508043f4a0185 P 7c69cc4810c74ffe8d9dfc25964557b5: Time spent starting tablet: real 0.013s user 0.010s sys 0.003s
W20260620 18:40:39.012725 9715 tablet.cc:2369] T 4cf1432d2825431093a508043f4a0185 P 335a53e810cc4fe4a7bc614a357e7db3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:40:39.013268 10007 raft_consensus.cc:482] T 4cf1432d2825431093a508043f4a0185 P 335a53e810cc4fe4a7bc614a357e7db3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:40:39.013626 10007 raft_consensus.cc:504] T 4cf1432d2825431093a508043f4a0185 P 335a53e810cc4fe4a7bc614a357e7db3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7fd899ef9a5456bba621fe7e5f5a1c8" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 43331 } } peers { permanent_uuid: "335a53e810cc4fe4a7bc614a357e7db3" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 42841 } } peers { permanent_uuid: "7c69cc4810c74ffe8d9dfc25964557b5" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 42891 } }
I20260620 18:40:39.015285 10007 leader_election.cc:288] T 4cf1432d2825431093a508043f4a0185 P 335a53e810cc4fe4a7bc614a357e7db3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e7fd899ef9a5456bba621fe7e5f5a1c8 (127.22.37.2:43331), 7c69cc4810c74ffe8d9dfc25964557b5 (127.22.37.3:42891)
I20260620 18:40:39.024643 9998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:39.015805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.1:47737 (local address 127.22.37.3:42891)
0620 18:40:39.016184 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:39.016190 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:39.016467 (+ 277us) server_negotiation.cc:407] Connection header received
0620 18:40:39.016758 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:39.016763 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:39.016858 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:39.016987 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:40:39.018031 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:39.019137 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:39.022123 (+ 2986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:39.023373 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:39.023507 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:39.023610 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:39.024254 (+ 644us) server_negotiation.cc:299] Negotiation successful
0620 18:40:39.024380 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219}
I20260620 18:40:39.024643 9996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:39.015750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.1:45043 (local address 127.22.37.2:43331)
0620 18:40:39.015989 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:39.015996 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:39.016842 (+ 846us) server_negotiation.cc:407] Connection header received
0620 18:40:39.017028 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:39.017035 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:39.017131 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:39.017208 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:40:39.018017 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:39.019137 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:39.022139 (+ 3002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:39.023381 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:39.023507 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:39.023605 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:39.024254 (+ 649us) server_negotiation.cc:299] Negotiation successful
0620 18:40:39.024380 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 18:40:39.024909 10009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:39.015603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.3:42891 (local address 127.22.37.1:47737)
0620 18:40:39.016351 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:39.016374 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 18:40:39.016612 (+ 238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:40:39.017352 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:40:39.017358 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:40:39.017469 (+ 111us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:40:39.017891 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:39.017904 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:39.019294 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:39.019299 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:39.021964 (+ 2665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:39.021979 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:39.023658 (+ 1679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:39.023663 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:39.023958 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:39.024118 (+ 160us) client_negotiation.cc:769] Sending connection context
0620 18:40:39.024216 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 18:40:39.024417 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"mutex_wait_us":447,"spinlock_wait_cycles":49920,"thread_start_us":388,"threads_started":1}
I20260620 18:40:39.024930 10010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:39.015682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.2:43331 (local address 127.22.37.1:45043)
0620 18:40:39.016711 (+ 1029us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:39.016729 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:40:39.016865 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:40:39.017267 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:40:39.017272 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:40:39.017302 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:40:39.017784 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:39.017797 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:39.019293 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:39.019299 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:39.021964 (+ 2665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:39.021979 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:39.023644 (+ 1665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:39.023651 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:39.023955 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:39.024115 (+ 160us) client_negotiation.cc:769] Sending connection context
0620 18:40:39.024216 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 18:40:39.024417 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":915,"mutex_wait_us":213,"thread_start_us":736,"threads_started":1}
I20260620 18:40:39.025965 9803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cf1432d2825431093a508043f4a0185" candidate_uuid: "335a53e810cc4fe4a7bc614a357e7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7fd899ef9a5456bba621fe7e5f5a1c8" is_pre_election: true
I20260620 18:40:39.025976 9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cf1432d2825431093a508043f4a0185" candidate_uuid: "335a53e810cc4fe4a7bc614a357e7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c69cc4810c74ffe8d9dfc25964557b5" is_pre_election: true
I20260620 18:40:39.026458 9937 raft_consensus.cc:2454] T 4cf1432d2825431093a508043f4a0185 P 7c69cc4810c74ffe8d9dfc25964557b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 335a53e810cc4fe4a7bc614a357e7db3 in term 0.
I20260620 18:40:39.026458 9803 raft_consensus.cc:2454] T 4cf1432d2825431093a508043f4a0185 P e7fd899ef9a5456bba621fe7e5f5a1c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 335a53e810cc4fe4a7bc614a357e7db3 in term 0.
I20260620 18:40:39.027122 9603 leader_election.cc:302] T 4cf1432d2825431093a508043f4a0185 P 335a53e810cc4fe4a7bc614a357e7db3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 335a53e810cc4fe4a7bc614a357e7db3, 7c69cc4810c74ffe8d9dfc25964557b5, e7fd899ef9a5456bba621fe7e5f5a1c8; no voters:
I20260620 18:40:39.027565 10007 raft_consensus.cc:2781] T 4cf1432d2825431093a508043f4a0185 P 335a53e810cc4fe4a7bc614a357e7db3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:40:39.027756 10007 raft_consensus.cc:482] T 4cf1432d2825431093a508043f4a0185 P 335a53e810cc4fe4a7bc614a357e7db3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:40:39.027859 10007 raft_consensus.cc:3037] T 4cf1432d2825431093a508043f4a0185 P 335a53e810cc4fe4a7bc614a357e7db3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:40:39.030261 10007 raft_consensus.cc:504] T 4cf1432d2825431093a508043f4a0185 P 335a53e810cc4fe4a7bc614a357e7db3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7fd899ef9a5456bba621fe7e5f5a1c8" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 43331 } } peers { permanent_uuid: "335a53e810cc4fe4a7bc614a357e7db3" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 42841 } } peers { permanent_uuid: "7c69cc4810c74ffe8d9dfc25964557b5" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 42891 } }
I20260620 18:40:39.030879 10007 leader_election.cc:288] T 4cf1432d2825431093a508043f4a0185 P 335a53e810cc4fe4a7bc614a357e7db3 [CANDIDATE]: Term 1 election: Requested vote from peers e7fd899ef9a5456bba621fe7e5f5a1c8 (127.22.37.2:43331), 7c69cc4810c74ffe8d9dfc25964557b5 (127.22.37.3:42891)
I20260620 18:40:39.031421 9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cf1432d2825431093a508043f4a0185" candidate_uuid: "335a53e810cc4fe4a7bc614a357e7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c69cc4810c74ffe8d9dfc25964557b5"
I20260620 18:40:39.031457 9803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cf1432d2825431093a508043f4a0185" candidate_uuid: "335a53e810cc4fe4a7bc614a357e7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7fd899ef9a5456bba621fe7e5f5a1c8"
I20260620 18:40:39.031670 9803 raft_consensus.cc:3037] T 4cf1432d2825431093a508043f4a0185 P e7fd899ef9a5456bba621fe7e5f5a1c8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:40:39.031679 9937 raft_consensus.cc:3037] T 4cf1432d2825431093a508043f4a0185 P 7c69cc4810c74ffe8d9dfc25964557b5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:40:39.033898 9803 raft_consensus.cc:2454] T 4cf1432d2825431093a508043f4a0185 P e7fd899ef9a5456bba621fe7e5f5a1c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 335a53e810cc4fe4a7bc614a357e7db3 in term 1.
I20260620 18:40:39.034253 9937 raft_consensus.cc:2454] T 4cf1432d2825431093a508043f4a0185 P 7c69cc4810c74ffe8d9dfc25964557b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 335a53e810cc4fe4a7bc614a357e7db3 in term 1.
I20260620 18:40:39.034302 9603 leader_election.cc:302] T 4cf1432d2825431093a508043f4a0185 P 335a53e810cc4fe4a7bc614a357e7db3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 335a53e810cc4fe4a7bc614a357e7db3, e7fd899ef9a5456bba621fe7e5f5a1c8; no voters:
I20260620 18:40:39.034596 10007 raft_consensus.cc:2781] T 4cf1432d2825431093a508043f4a0185 P 335a53e810cc4fe4a7bc614a357e7db3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:40:39.035075 10007 raft_consensus.cc:686] T 4cf1432d2825431093a508043f4a0185 P 335a53e810cc4fe4a7bc614a357e7db3 [term 1 LEADER]: Becoming Leader. State: Replica: 335a53e810cc4fe4a7bc614a357e7db3, State: Running, Role: LEADER
I20260620 18:40:39.035557 10007 consensus_queue.cc:237] T 4cf1432d2825431093a508043f4a0185 P 335a53e810cc4fe4a7bc614a357e7db3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7fd899ef9a5456bba621fe7e5f5a1c8" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 43331 } } peers { permanent_uuid: "335a53e810cc4fe4a7bc614a357e7db3" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 42841 } } peers { permanent_uuid: "7c69cc4810c74ffe8d9dfc25964557b5" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 42891 } }
I20260620 18:40:39.040478 9524 catalog_manager.cc:5495] T 4cf1432d2825431093a508043f4a0185 P 335a53e810cc4fe4a7bc614a357e7db3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 335a53e810cc4fe4a7bc614a357e7db3 (127.22.37.1). New cstate: current_term: 1 leader_uuid: "335a53e810cc4fe4a7bc614a357e7db3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7fd899ef9a5456bba621fe7e5f5a1c8" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 43331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "335a53e810cc4fe4a7bc614a357e7db3" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 42841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7c69cc4810c74ffe8d9dfc25964557b5" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 42891 } health_report { overall_health: UNKNOWN } } }
I20260620 18:40:39.059224 9999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:39.053449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45460 (local address 127.22.37.1:42841)
0620 18:40:39.053663 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:39.053670 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:39.053777 (+ 107us) server_negotiation.cc:407] Connection header received
0620 18:40:39.054048 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:39.054052 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:39.054146 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:39.054297 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:40:39.054669 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:39.055393 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:39.056986 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:39.057509 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:39.057633 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:39.058142 (+ 509us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:40:39.058654 (+ 512us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:40:39.058816 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:39.058867 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 18:40:39.058947 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260620 18:40:39.066808 9849 tablet.cc:2369] T 4cf1432d2825431093a508043f4a0185 P e7fd899ef9a5456bba621fe7e5f5a1c8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:40:39.067464 9937 raft_consensus.cc:1261] T 4cf1432d2825431093a508043f4a0185 P 7c69cc4810c74ffe8d9dfc25964557b5 [term 1 FOLLOWER]: Refusing update from remote peer 335a53e810cc4fe4a7bc614a357e7db3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:40:39.067520 9803 raft_consensus.cc:1261] T 4cf1432d2825431093a508043f4a0185 P e7fd899ef9a5456bba621fe7e5f5a1c8 [term 1 FOLLOWER]: Refusing update from remote peer 335a53e810cc4fe4a7bc614a357e7db3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:40:39.068537 10011 consensus_queue.cc:1035] T 4cf1432d2825431093a508043f4a0185 P 335a53e810cc4fe4a7bc614a357e7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c69cc4810c74ffe8d9dfc25964557b5" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 42891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:40:39.068873 10007 consensus_queue.cc:1035] T 4cf1432d2825431093a508043f4a0185 P 335a53e810cc4fe4a7bc614a357e7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7fd899ef9a5456bba621fe7e5f5a1c8" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 43331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 18:40:39.074162 9983 tablet.cc:2369] T 4cf1432d2825431093a508043f4a0185 P 7c69cc4810c74ffe8d9dfc25964557b5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:40:39.094033 9523 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39644:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:40:39.094390 9523 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39644:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:40:39.098557 9523 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1cffb780af35489381efe222d20ff2b3: Sending DeleteTablet for 3 replicas of tablet 4cf1432d2825431093a508043f4a0185
I20260620 18:40:39.099550 9917 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cf1432d2825431093a508043f4a0185 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:40:39 UTC) from {username='kudu', principal='kudu/127.22.37.62@KRBTEST.COM'} at 127.0.0.1:53196
I20260620 18:40:39.099556 9783 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cf1432d2825431093a508043f4a0185 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:40:39 UTC) from {username='kudu', principal='kudu/127.22.37.62@KRBTEST.COM'} at 127.0.0.1:47996
I20260620 18:40:39.100404 9649 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cf1432d2825431093a508043f4a0185 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:40:39 UTC) from {username='kudu', principal='kudu/127.22.37.62@KRBTEST.COM'} at 127.0.0.1:45444
I20260620 18:40:39.100739 9916 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cf1432d2825431093a508043f4a0185 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:40:39 UTC) from {username='kudu', principal='kudu/127.22.37.62@KRBTEST.COM'} at 127.0.0.1:53196
I20260620 18:40:39.102454 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 9586
I20260620 18:40:39.112970 10025 tablet_replica.cc:330] T 4cf1432d2825431093a508043f4a0185 P e7fd899ef9a5456bba621fe7e5f5a1c8: stopping tablet replica
I20260620 18:40:39.113786 10024 tablet_replica.cc:330] T 4cf1432d2825431093a508043f4a0185 P 7c69cc4810c74ffe8d9dfc25964557b5: stopping tablet replica
I20260620 18:40:39.114718 10024 raft_consensus.cc:2229] T 4cf1432d2825431093a508043f4a0185 P 7c69cc4810c74ffe8d9dfc25964557b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:40:39.115306 10025 raft_consensus.cc:2229] T 4cf1432d2825431093a508043f4a0185 P e7fd899ef9a5456bba621fe7e5f5a1c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:40:39.115962 10024 raft_consensus.cc:2258] T 4cf1432d2825431093a508043f4a0185 P 7c69cc4810c74ffe8d9dfc25964557b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:40:39.115985 10026 tablet_replica.cc:330] T 4cf1432d2825431093a508043f4a0185 P 335a53e810cc4fe4a7bc614a357e7db3: stopping tablet replica
I20260620 18:40:39.116457 10025 raft_consensus.cc:2258] T 4cf1432d2825431093a508043f4a0185 P e7fd899ef9a5456bba621fe7e5f5a1c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:40:39.116503 10026 raft_consensus.cc:2229] T 4cf1432d2825431093a508043f4a0185 P 335a53e810cc4fe4a7bc614a357e7db3 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:40:39.117460 10026 raft_consensus.cc:2258] T 4cf1432d2825431093a508043f4a0185 P 335a53e810cc4fe4a7bc614a357e7db3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:40:39.119228 10024 ts_tablet_manager.cc:1905] T 4cf1432d2825431093a508043f4a0185 P 7c69cc4810c74ffe8d9dfc25964557b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:40:39.119228 10025 ts_tablet_manager.cc:1905] T 4cf1432d2825431093a508043f4a0185 P e7fd899ef9a5456bba621fe7e5f5a1c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:40:39.126426 10024 ts_tablet_manager.cc:1918] T 4cf1432d2825431093a508043f4a0185 P 7c69cc4810c74ffe8d9dfc25964557b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:40:39.126427 10025 ts_tablet_manager.cc:1918] T 4cf1432d2825431093a508043f4a0185 P e7fd899ef9a5456bba621fe7e5f5a1c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:40:39.126600 10024 log.cc:1196] T 4cf1432d2825431093a508043f4a0185 P 7c69cc4810c74ffe8d9dfc25964557b5: Deleting WAL directory at /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980527431848-22676-0/minicluster-data/ts-2/wal/wals/4cf1432d2825431093a508043f4a0185
I20260620 18:40:39.126770 10025 log.cc:1196] T 4cf1432d2825431093a508043f4a0185 P e7fd899ef9a5456bba621fe7e5f5a1c8: Deleting WAL directory at /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980527431848-22676-0/minicluster-data/ts-1/wal/wals/4cf1432d2825431093a508043f4a0185
I20260620 18:40:39.127164 10024 ts_tablet_manager.cc:1939] T 4cf1432d2825431093a508043f4a0185 P 7c69cc4810c74ffe8d9dfc25964557b5: Deleting consensus metadata
I20260620 18:40:39.127482 10025 ts_tablet_manager.cc:1939] T 4cf1432d2825431093a508043f4a0185 P e7fd899ef9a5456bba621fe7e5f5a1c8: Deleting consensus metadata
Jun 20 18:40:39 dist-test-slave-5lgp krb5kdc[9480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.37.1@KRBTEST.COM
I20260620 18:40:39.129216 9511 catalog_manager.cc:4841] TS 7c69cc4810c74ffe8d9dfc25964557b5 (127.22.37.3:42891): tablet 4cf1432d2825431093a508043f4a0185 (table test-table [id=9c40bfa28ac6443289a4e9186cbca458]) successfully deleted
I20260620 18:40:39.129505 9510 catalog_manager.cc:4841] TS e7fd899ef9a5456bba621fe7e5f5a1c8 (127.22.37.2:43331): tablet 4cf1432d2825431093a508043f4a0185 (table test-table [id=9c40bfa28ac6443289a4e9186cbca458]) successfully deleted
W20260620 18:40:39.130632 9511 catalog_manager.cc:4808] TS 7c69cc4810c74ffe8d9dfc25964557b5 (127.22.37.3:42891): delete failed for tablet 4cf1432d2825431093a508043f4a0185 because the tablet was not found. No further retry: Not found: Tablet not found: 4cf1432d2825431093a508043f4a0185
I20260620 18:40:39.132010 10026 ts_tablet_manager.cc:1905] T 4cf1432d2825431093a508043f4a0185 P 335a53e810cc4fe4a7bc614a357e7db3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:40:39.138679 10026 ts_tablet_manager.cc:1918] T 4cf1432d2825431093a508043f4a0185 P 335a53e810cc4fe4a7bc614a357e7db3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:40:39.139205 9999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:39.104322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45462 (local address 127.22.37.1:42841)
0620 18:40:39.104496 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:39.104502 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:39.117457 (+ 12955us) server_negotiation.cc:407] Connection header received
0620 18:40:39.117935 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:39.117944 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:39.118037 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:39.118114 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:40:39.120326 (+ 2212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:39.121602 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:39.123874 (+ 2272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:39.124586 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:39.124684 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:39.130980 (+ 6296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:40:39.130996 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:40:39.131015 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:40:39.131082 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:40:39.135389 (+ 4307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:40:39.136473 (+ 1084us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:40:39.136482 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:40:39.136486 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:40:39.136674 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:40:39.137149 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:40:39.137154 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:40:39.137157 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:40:39.137681 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:40:39.137932 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:39.138564 (+ 632us) server_negotiation.cc:299] Negotiation successful
0620 18:40:39.138817 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 18:40:39.138855 10026 log.cc:1196] T 4cf1432d2825431093a508043f4a0185 P 335a53e810cc4fe4a7bc614a357e7db3: Deleting WAL directory at /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980527431848-22676-0/minicluster-data/ts-0/wal/wals/4cf1432d2825431093a508043f4a0185
I20260620 18:40:39.139902 10026 ts_tablet_manager.cc:1939] T 4cf1432d2825431093a508043f4a0185 P 335a53e810cc4fe4a7bc614a357e7db3: Deleting consensus metadata
I20260620 18:40:39.141271 9709 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:40:39.141870 9512 catalog_manager.cc:4841] TS 335a53e810cc4fe4a7bc614a357e7db3 (127.22.37.1:42841): tablet 4cf1432d2825431093a508043f4a0185 (table test-table [id=9c40bfa28ac6443289a4e9186cbca458]) successfully deleted
I20260620 18:40:39.252954 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 9586
I20260620 18:40:39.271641 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 9720
Jun 20 18:40:39 dist-test-slave-5lgp krb5kdc[9480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.37.2@KRBTEST.COM
I20260620 18:40:39.286645 9996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:39.272423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48002 (local address 127.22.37.2:43331)
0620 18:40:39.272603 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:39.272607 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:39.272624 (+ 17us) server_negotiation.cc:407] Connection header received
0620 18:40:39.272679 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:39.272683 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:39.272745 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:39.272838 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:40:39.274127 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:39.274884 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:39.276118 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:39.276510 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:39.276605 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:39.281315 (+ 4710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:40:39.281330 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:40:39.281347 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:40:39.281449 (+ 102us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:40:39.284181 (+ 2732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:40:39.284840 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:40:39.284846 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:40:39.284849 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:40:39.284992 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:40:39.285308 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:40:39.285311 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:40:39.285313 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:40:39.285753 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:40:39.286016 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:39.286213 (+ 197us) server_negotiation.cc:299] Negotiation successful
0620 18:40:39.286378 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 18:40:39.287352 9843 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:40:39.377336 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 9720
I20260620 18:40:39.394153 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 9854
Jun 20 18:40:39 dist-test-slave-5lgp krb5kdc[9480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.37.3@KRBTEST.COM
I20260620 18:40:39.409641 9998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:39.394954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53200 (local address 127.22.37.3:42891)
0620 18:40:39.395126 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:39.395131 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:39.395150 (+ 19us) server_negotiation.cc:407] Connection header received
0620 18:40:39.395306 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:39.395310 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:39.395383 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:39.395464 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:40:39.396899 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:39.397744 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:39.399040 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:39.399439 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:39.399544 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:39.404046 (+ 4502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:40:39.404062 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:40:39.404078 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:40:39.404113 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:40:39.406980 (+ 2867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:40:39.407661 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:40:39.407669 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:40:39.407671 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:40:39.407812 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:40:39.408186 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:40:39.408190 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:40:39.408192 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:40:39.408552 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:40:39.408775 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:39.409071 (+ 296us) server_negotiation.cc:299] Negotiation successful
0620 18:40:39.409244 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 18:40:39.410301 9977 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:40:39.511339 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 9854
I20260620 18:40:39.529620 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 9493
I20260620 18:40:39.530853 9555 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:40:39.641834 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 9493
2026-06-20T18:40:39Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4253 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-20T18:40:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:40:39Z Disabled control of system clock
I20260620 18:40:39.717665 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980527431848-22676-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980527431848-22676-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980527431848-22676-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.62:35347
--webserver_interface=127.22.37.62
--webserver_port=0
--builtin_ntp_servers=127.22.37.20:44041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.37.62:35347
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 18:40:40.042412 10035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:40:40.042695 10035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:40:40.042780 10035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:40:40.050027 10035 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:40:40.050133 10035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:40:40.050171 10035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:40:40.050220 10035 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:40:40.050266 10035 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:40:40.059504 10035 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:44041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980527431848-22676-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980527431848-22676-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.37.62:35347
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.62:35347
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.37.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980527431848-22676-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:40:40.060777 10035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:40:40.062577 10035 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:40:40.074605 10040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:40:40.074671 10041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:40:40.074647 10043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:40:40.075467 10035 server_base.cc:1034] running on GCE node
I20260620 18:40:40.076304 10035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:40:40.077776 10035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:40:40.079008 10035 hybrid_clock.cc:648] HybridClock initialized: now 1781980840078959 us; error 62 us; skew 500 ppm
I20260620 18:40:40.079353 10035 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:40:40.081604 10035 webserver.cc:460] Webserver started at http://127.22.37.62:46349/ using document root <none> and password file <none>
I20260620 18:40:40.082289 10035 fs_manager.cc:359] Metadata directory not provided
I20260620 18:40:40.082422 10035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:40:40.082743 10035 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:40:40.085714 10035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980527431848-22676-0/minicluster-data/master-0/data/instance:
uuid: "5a2ccdf5aa55489dbba20dcaacb2b3c4"
format_stamp: "Formatted at 2026-06-20 18:40:40 on dist-test-slave-5lgp"
server_key: "e2c9a1e92e8d1c0bcb8b24d722e8e637"
server_key_iv: "03e38fdcc53ab73c3afc4e03e702eab8"
server_key_version: "encryptionkey@0"
I20260620 18:40:40.086524 10035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980527431848-22676-0/minicluster-data/master-0/wal/instance:
uuid: "5a2ccdf5aa55489dbba20dcaacb2b3c4"
format_stamp: "Formatted at 2026-06-20 18:40:40 on dist-test-slave-5lgp"
server_key: "e2c9a1e92e8d1c0bcb8b24d722e8e637"
server_key_iv: "03e38fdcc53ab73c3afc4e03e702eab8"
server_key_version: "encryptionkey@0"
I20260620 18:40:40.091504 10035 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260620 18:40:40.094944 10049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:40:40.096195 10035 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 18:40:40.096376 10035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980527431848-22676-0/minicluster-data/master-0/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980527431848-22676-0/minicluster-data/master-0/wal
uuid: "5a2ccdf5aa55489dbba20dcaacb2b3c4"
format_stamp: "Formatted at 2026-06-20 18:40:40 on dist-test-slave-5lgp"
server_key: "e2c9a1e92e8d1c0bcb8b24d722e8e637"
server_key_iv: "03e38fdcc53ab73c3afc4e03e702eab8"
server_key_version: "encryptionkey@0"
I20260620 18:40:40.096585 10035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980527431848-22676-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980527431848-22676-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980527431848-22676-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:40:40.115000 10035 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:40:40.115789 10035 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:40:40.116082 10035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:40:40.134464 10035 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.62:35347
I20260620 18:40:40.134500 10100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.62:35347 every 8 connection(s)
I20260620 18:40:40.135794 10035 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
I20260620 18:40:40.138684 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 10035
I20260620 18:40:40.138873 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980527431848-22676-0/minicluster-data/master-0/wal/instance
I20260620 18:40:40.139407 22676 external_mini_cluster.cc:1428] Setting key c8e38bc304a73621e1a10efd08c2cc1d
I20260620 18:40:40.140012 10101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:40:40.152372 10101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a2ccdf5aa55489dbba20dcaacb2b3c4: Bootstrap starting.
I20260620 18:40:40.153146 10104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:40.141359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37910 (local address 127.22.37.62:35347)
0620 18:40:40.142338 (+ 979us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:40.142349 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:40.142390 (+ 41us) server_negotiation.cc:407] Connection header received
0620 18:40:40.144636 (+ 2246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:40.144659 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:40.145134 (+ 475us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:40.145584 (+ 450us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:40:40.146151 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:40.147450 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:40.148759 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:40.149547 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:40.149780 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:40.150881 (+ 1101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:40:40.150888 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:40:40.150900 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:40:40.151422 (+ 522us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:40:40.151486 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:40:40.151508 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:40:40.151626 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:40.151850 (+ 224us) server_negotiation.cc:299] Negotiation successful
0620 18:40:40.152136 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":331,"threads_started":1}
I20260620 18:40:40.155494 10101 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a2ccdf5aa55489dbba20dcaacb2b3c4: Neither blocks nor log segments found. Creating new log.
I20260620 18:40:40.156535 10101 log.cc:824] T 00000000000000000000000000000000 P 5a2ccdf5aa55489dbba20dcaacb2b3c4: Log is configured to *not* fsync() on all Append() calls
I20260620 18:40:40.159575 10101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a2ccdf5aa55489dbba20dcaacb2b3c4: No bootstrap required, opened a new log
I20260620 18:40:40.167327 10101 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a2ccdf5aa55489dbba20dcaacb2b3c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a2ccdf5aa55489dbba20dcaacb2b3c4" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 35347 } }
I20260620 18:40:40.167685 10101 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5a2ccdf5aa55489dbba20dcaacb2b3c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:40:40.167783 10101 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a2ccdf5aa55489dbba20dcaacb2b3c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a2ccdf5aa55489dbba20dcaacb2b3c4, State: Initialized, Role: FOLLOWER
I20260620 18:40:40.168318 10101 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a2ccdf5aa55489dbba20dcaacb2b3c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a2ccdf5aa55489dbba20dcaacb2b3c4" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 35347 } }
I20260620 18:40:40.168535 10101 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5a2ccdf5aa55489dbba20dcaacb2b3c4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:40:40.168663 10101 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a2ccdf5aa55489dbba20dcaacb2b3c4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:40:40.168828 10101 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a2ccdf5aa55489dbba20dcaacb2b3c4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:40:40.171023 10101 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a2ccdf5aa55489dbba20dcaacb2b3c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a2ccdf5aa55489dbba20dcaacb2b3c4" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 35347 } }
I20260620 18:40:40.171504 10101 leader_election.cc:302] T 00000000000000000000000000000000 P 5a2ccdf5aa55489dbba20dcaacb2b3c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a2ccdf5aa55489dbba20dcaacb2b3c4; no voters:
I20260620 18:40:40.172206 10101 leader_election.cc:288] T 00000000000000000000000000000000 P 5a2ccdf5aa55489dbba20dcaacb2b3c4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:40:40.172353 10106 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a2ccdf5aa55489dbba20dcaacb2b3c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:40:40.173167 10106 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a2ccdf5aa55489dbba20dcaacb2b3c4 [term 1 LEADER]: Becoming Leader. State: Replica: 5a2ccdf5aa55489dbba20dcaacb2b3c4, State: Running, Role: LEADER
I20260620 18:40:40.173713 10106 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a2ccdf5aa55489dbba20dcaacb2b3c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a2ccdf5aa55489dbba20dcaacb2b3c4" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 35347 } }
I20260620 18:40:40.174433 10101 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a2ccdf5aa55489dbba20dcaacb2b3c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:40:40.176667 10107 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a2ccdf5aa55489dbba20dcaacb2b3c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5a2ccdf5aa55489dbba20dcaacb2b3c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a2ccdf5aa55489dbba20dcaacb2b3c4" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 35347 } } }
I20260620 18:40:40.177147 10107 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a2ccdf5aa55489dbba20dcaacb2b3c4 [sys.catalog]: This master's current role is: LEADER
I20260620 18:40:40.177628 10108 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a2ccdf5aa55489dbba20dcaacb2b3c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a2ccdf5aa55489dbba20dcaacb2b3c4. Latest consensus state: current_term: 1 leader_uuid: "5a2ccdf5aa55489dbba20dcaacb2b3c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a2ccdf5aa55489dbba20dcaacb2b3c4" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 35347 } } }
I20260620 18:40:40.178066 10108 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a2ccdf5aa55489dbba20dcaacb2b3c4 [sys.catalog]: This master's current role is: LEADER
I20260620 18:40:40.181067 10115 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:40:40.186374 10115 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:40:40.194869 10115 catalog_manager.cc:1347] Generated new cluster ID: 9544d42fc106412cbe0c7bff64b7131b
I20260620 18:40:40.194979 10115 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:40:40.218525 10115 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:40:40.220309 10115 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:40:40.230044 10115 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5a2ccdf5aa55489dbba20dcaacb2b3c4: Generated new TSK 0
I20260620 18:40:40.231081 10115 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:40:40.242417 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980527431848-22676-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980527431848-22676-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980527431848-22676-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980527431848-22676-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.37.1:0
--local_ip_for_outbound_sockets=127.22.37.1
--webserver_interface=127.22.37.1
--webserver_port=0
--tserver_master_addrs=127.22.37.62:35347
--builtin_ntp_servers=127.22.37.20:44041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 18:40:40.585610 10125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:40:40.585881 10125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:40:40.585964 10125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:40:40.592988 10125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:40:40.593108 10125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:40:40.593212 10125 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.37.1
I20260620 18:40:40.602754 10125 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:44041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980527431848-22676-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980527431848-22676-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.37.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980527431848-22676-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.37.1
--webserver_port=0
--tserver_master_addrs=127.22.37.62:35347
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.37.1
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980527431848-22676-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:40:40.603986 10125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:40:40.605861 10125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:40:40.619163 10130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:40:40.619251 10131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:40:40.619212 10133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:40:40.620893 10125 server_base.cc:1034] running on GCE node
I20260620 18:40:40.621618 10125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:40:40.622622 10125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:40:40.623832 10125 hybrid_clock.cc:648] HybridClock initialized: now 1781980840623778 us; error 40 us; skew 500 ppm
I20260620 18:40:40.624163 10125 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:40:40.630661 10125 webserver.cc:460] Webserver started at http://127.22.37.1:35541/ using document root <none> and password file <none>
I20260620 18:40:40.631409 10125 fs_manager.cc:359] Metadata directory not provided
I20260620 18:40:40.631536 10125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:40:40.631873 10125 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:40:40.634922 10125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980527431848-22676-0/minicluster-data/ts-0/data/instance:
uuid: "3abf467f21b74971a4b252a8fe60627a"
format_stamp: "Formatted at 2026-06-20 18:40:40 on dist-test-slave-5lgp"
server_key: "f1b71c3248335c1f494094ca189977d0"
server_key_iv: "5b04ca9e6c8bf281f4f7d9689b089527"
server_key_version: "encryptionkey@0"
I20260620 18:40:40.635736 10125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980527431848-22676-0/minicluster-data/ts-0/wal/instance:
uuid: "3abf467f21b74971a4b252a8fe60627a"
format_stamp: "Formatted at 2026-06-20 18:40:40 on dist-test-slave-5lgp"
server_key: "f1b71c3248335c1f494094ca189977d0"
server_key_iv: "5b04ca9e6c8bf281f4f7d9689b089527"
server_key_version: "encryptionkey@0"
I20260620 18:40:40.641100 10125 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260620 18:40:40.644806 10139 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:40:40.646291 10125 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:40:40.646490 10125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980527431848-22676-0/minicluster-data/ts-0/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980527431848-22676-0/minicluster-data/ts-0/wal
uuid: "3abf467f21b74971a4b252a8fe60627a"
format_stamp: "Formatted at 2026-06-20 18:40:40 on dist-test-slave-5lgp"
server_key: "f1b71c3248335c1f494094ca189977d0"
server_key_iv: "5b04ca9e6c8bf281f4f7d9689b089527"
server_key_version: "encryptionkey@0"
I20260620 18:40:40.646751 10125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980527431848-22676-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980527431848-22676-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980527431848-22676-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:40:40.659682 10125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:40:40.660606 10125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:40:40.660930 10125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:40:40.662029 10125 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:40:40.663512 10125 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:40:40.663645 10125 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:40:40.663779 10125 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:40:40.663870 10125 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:40:40.701330 10125 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.1:36117
I20260620 18:40:40.701402 10251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.1:36117 every 8 connection(s)
I20260620 18:40:40.702807 10125 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980527431848-22676-0/minicluster-data/ts-0/data/info.pb
I20260620 18:40:40.707352 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 10125
I20260620 18:40:40.707579 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980527431848-22676-0/minicluster-data/ts-0/wal/instance
I20260620 18:40:40.708056 22676 external_mini_cluster.cc:1428] Setting key db9d361862197635636abee032b35dfa
I20260620 18:40:40.711273 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980527431848-22676-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980527431848-22676-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980527431848-22676-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980527431848-22676-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.37.2:0
--local_ip_for_outbound_sockets=127.22.37.2
--webserver_interface=127.22.37.2
--webserver_port=0
--tserver_master_addrs=127.22.37.62:35347
--builtin_ntp_servers=127.22.37.20:44041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 18:40:40.718132 10254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:40.705552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.1:43077 (local address 127.22.37.62:35347)
0620 18:40:40.706047 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:40.706053 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:40.707310 (+ 1257us) server_negotiation.cc:407] Connection header received
0620 18:40:40.709477 (+ 2167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:40.709486 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:40.709584 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:40.709709 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:40:40.711282 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:40.712131 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:40.714805 (+ 2674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:40.715224 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:40.715324 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:40.717190 (+ 1866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:40:40.717196 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:40:40.717200 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:40:40.717222 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:40:40.717266 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:40:40.717279 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:40:40.717358 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:40.717718 (+ 360us) server_negotiation.cc:299] Negotiation successful
0620 18:40:40.717814 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":219,"threads_started":1}
I20260620 18:40:40.719683 10255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:40.706202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:35347 (local address 127.22.37.1:43077)
0620 18:40:40.707138 (+ 936us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:40.707174 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 18:40:40.709161 (+ 1987us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:40:40.709970 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:40:40.709979 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:40:40.710131 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:40:40.711019 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:40.711051 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:40.712309 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:40.712318 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:40.714623 (+ 2305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:40.714642 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:40.715475 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:40.715484 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:40.715874 (+ 390us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:40.716744 (+ 870us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:40:40.716750 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:40:40.716815 (+ 65us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:40:40.716945 (+ 130us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:40:40.717422 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:40:40.717437 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:40:40.717449 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 18:40:40.717665 (+ 216us) client_negotiation.cc:240] Negotiation successful
0620 18:40:40.718152 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":286,"threads_started":1}
I20260620 18:40:40.721760 10252 heartbeater.cc:347] Connected to a master server at 127.22.37.62:35347
I20260620 18:40:40.722095 10252 heartbeater.cc:464] Registering TS with master...
I20260620 18:40:40.722903 10252 heartbeater.cc:511] Master 127.22.37.62:35347 requested a full tablet report, sending...
I20260620 18:40:40.725206 10066 ts_manager.cc:194] Registered new tserver with Master: 3abf467f21b74971a4b252a8fe60627a (127.22.37.1:36117)
I20260620 18:40:40.726912 10066 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.37.1:43077
W20260620 18:40:41.046794 10256 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:40:41.047122 10256 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:40:41.047286 10256 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:40:41.054701 10256 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:40:41.054841 10256 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:40:41.055024 10256 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.37.2
I20260620 18:40:41.064762 10256 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:44041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980527431848-22676-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980527431848-22676-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.37.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980527431848-22676-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.37.2
--webserver_port=0
--tserver_master_addrs=127.22.37.62:35347
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.37.2
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980527431848-22676-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:40:41.066171 10256 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:40:41.067925 10256 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:40:41.081923 10261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:40:41.081969 10264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:40:41.081993 10262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:40:41.082762 10256 server_base.cc:1034] running on GCE node
I20260620 18:40:41.083453 10256 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:40:41.084427 10256 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:40:41.085635 10256 hybrid_clock.cc:648] HybridClock initialized: now 1781980841085585 us; error 47 us; skew 500 ppm
I20260620 18:40:41.085985 10256 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:40:41.088327 10256 webserver.cc:460] Webserver started at http://127.22.37.2:37741/ using document root <none> and password file <none>
I20260620 18:40:41.089033 10256 fs_manager.cc:359] Metadata directory not provided
I20260620 18:40:41.089149 10256 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:40:41.089540 10256 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:40:41.092316 10256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980527431848-22676-0/minicluster-data/ts-1/data/instance:
uuid: "abbe673979ae4b509c91573c23599e36"
format_stamp: "Formatted at 2026-06-20 18:40:41 on dist-test-slave-5lgp"
server_key: "abec924ea08459611f4bac7eb6c3693f"
server_key_iv: "7847c0fb09b09c7eb395bdebc46c176b"
server_key_version: "encryptionkey@0"
I20260620 18:40:41.093093 10256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980527431848-22676-0/minicluster-data/ts-1/wal/instance:
uuid: "abbe673979ae4b509c91573c23599e36"
format_stamp: "Formatted at 2026-06-20 18:40:41 on dist-test-slave-5lgp"
server_key: "abec924ea08459611f4bac7eb6c3693f"
server_key_iv: "7847c0fb09b09c7eb395bdebc46c176b"
server_key_version: "encryptionkey@0"
I20260620 18:40:41.098016 10256 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 18:40:41.101531 10270 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:40:41.102854 10256 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 18:40:41.103025 10256 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980527431848-22676-0/minicluster-data/ts-1/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980527431848-22676-0/minicluster-data/ts-1/wal
uuid: "abbe673979ae4b509c91573c23599e36"
format_stamp: "Formatted at 2026-06-20 18:40:41 on dist-test-slave-5lgp"
server_key: "abec924ea08459611f4bac7eb6c3693f"
server_key_iv: "7847c0fb09b09c7eb395bdebc46c176b"
server_key_version: "encryptionkey@0"
I20260620 18:40:41.103233 10256 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980527431848-22676-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980527431848-22676-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980527431848-22676-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:40:41.121554 10256 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:40:41.122435 10256 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:40:41.122727 10256 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:40:41.123762 10256 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:40:41.125244 10256 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:40:41.125380 10256 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:40:41.125527 10256 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:40:41.125622 10256 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:40:41.172281 10256 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.2:38691
I20260620 18:40:41.172328 10382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.2:38691 every 8 connection(s)
I20260620 18:40:41.173715 10256 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980527431848-22676-0/minicluster-data/ts-1/data/info.pb
I20260620 18:40:41.180722 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 10256
I20260620 18:40:41.180910 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980527431848-22676-0/minicluster-data/ts-1/wal/instance
I20260620 18:40:41.181344 22676 external_mini_cluster.cc:1428] Setting key 81c6b8648aae734b356186549ce94315
I20260620 18:40:41.184100 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980527431848-22676-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980527431848-22676-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980527431848-22676-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980527431848-22676-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.37.3:0
--local_ip_for_outbound_sockets=127.22.37.3
--webserver_interface=127.22.37.3
--webserver_port=0
--tserver_master_addrs=127.22.37.62:35347
--builtin_ntp_servers=127.22.37.20:44041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 18:40:41.187047 10254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:41.176468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.2:42525 (local address 127.22.37.62:35347)
0620 18:40:41.176652 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:41.176656 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:41.178241 (+ 1585us) server_negotiation.cc:407] Connection header received
0620 18:40:41.180036 (+ 1795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:41.180042 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:41.180144 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:41.180257 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:40:41.181545 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:41.182422 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:41.184138 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:41.184672 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:41.184801 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:41.186248 (+ 1447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:40:41.186253 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:40:41.186256 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:40:41.186275 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:40:41.186306 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:40:41.186317 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:40:41.186410 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:41.186647 (+ 237us) server_negotiation.cc:299] Negotiation successful
0620 18:40:41.186743 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 18:40:41.188143 10385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:41.177054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:35347 (local address 127.22.37.2:42525)
0620 18:40:41.178072 (+ 1018us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:41.178116 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 18:40:41.179765 (+ 1649us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:40:41.180505 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:40:41.180514 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:40:41.180613 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:40:41.181308 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:41.181326 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:41.182567 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:41.182572 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:41.183988 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:41.183999 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:41.184902 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:41.184910 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:41.185138 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:41.185908 (+ 770us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:40:41.185911 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:40:41.185952 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:40:41.186036 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:40:41.186409 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:40:41.186423 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:40:41.186433 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 18:40:41.186623 (+ 190us) client_negotiation.cc:240] Negotiation successful
0620 18:40:41.187025 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":315,"threads_started":1}
I20260620 18:40:41.189550 10383 heartbeater.cc:347] Connected to a master server at 127.22.37.62:35347
I20260620 18:40:41.189882 10383 heartbeater.cc:464] Registering TS with master...
I20260620 18:40:41.190795 10383 heartbeater.cc:511] Master 127.22.37.62:35347 requested a full tablet report, sending...
I20260620 18:40:41.192471 10066 ts_manager.cc:194] Registered new tserver with Master: abbe673979ae4b509c91573c23599e36 (127.22.37.2:38691)
I20260620 18:40:41.193495 10066 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.37.2:42525
W20260620 18:40:41.515166 10386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:40:41.515439 10386 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:40:41.515523 10386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:40:41.522728 10386 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:40:41.522821 10386 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:40:41.522953 10386 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.37.3
I20260620 18:40:41.532114 10386 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:44041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980527431848-22676-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980527431848-22676-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.37.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980527431848-22676-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.37.3
--webserver_port=0
--tserver_master_addrs=127.22.37.62:35347
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.37.3
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980527431848-22676-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:40:41.533442 10386 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:40:41.535200 10386 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:40:41.548168 10391 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:40:41.548218 10394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:40:41.548264 10392 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:40:41.549053 10386 server_base.cc:1034] running on GCE node
I20260620 18:40:41.549749 10386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:40:41.550819 10386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:40:41.552016 10386 hybrid_clock.cc:648] HybridClock initialized: now 1781980841551980 us; error 35 us; skew 500 ppm
I20260620 18:40:41.552347 10386 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:40:41.554591 10386 webserver.cc:460] Webserver started at http://127.22.37.3:43691/ using document root <none> and password file <none>
I20260620 18:40:41.555295 10386 fs_manager.cc:359] Metadata directory not provided
I20260620 18:40:41.555413 10386 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:40:41.555743 10386 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:40:41.558671 10386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980527431848-22676-0/minicluster-data/ts-2/data/instance:
uuid: "ce510824a42a4c0eb11b9ddb0b1937bb"
format_stamp: "Formatted at 2026-06-20 18:40:41 on dist-test-slave-5lgp"
server_key: "2101a6a7bb67615ef86db68b3b7a1581"
server_key_iv: "e293d54d5a61ffbebc6176bbdd6c8d76"
server_key_version: "encryptionkey@0"
I20260620 18:40:41.559429 10386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980527431848-22676-0/minicluster-data/ts-2/wal/instance:
uuid: "ce510824a42a4c0eb11b9ddb0b1937bb"
format_stamp: "Formatted at 2026-06-20 18:40:41 on dist-test-slave-5lgp"
server_key: "2101a6a7bb67615ef86db68b3b7a1581"
server_key_iv: "e293d54d5a61ffbebc6176bbdd6c8d76"
server_key_version: "encryptionkey@0"
I20260620 18:40:41.564368 10386 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 18:40:41.567710 10400 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:40:41.569062 10386 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:40:41.569255 10386 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980527431848-22676-0/minicluster-data/ts-2/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980527431848-22676-0/minicluster-data/ts-2/wal
uuid: "ce510824a42a4c0eb11b9ddb0b1937bb"
format_stamp: "Formatted at 2026-06-20 18:40:41 on dist-test-slave-5lgp"
server_key: "2101a6a7bb67615ef86db68b3b7a1581"
server_key_iv: "e293d54d5a61ffbebc6176bbdd6c8d76"
server_key_version: "encryptionkey@0"
I20260620 18:40:41.569484 10386 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980527431848-22676-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980527431848-22676-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980527431848-22676-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:40:41.593438 10386 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:40:41.594345 10386 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:40:41.594650 10386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:40:41.595695 10386 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:40:41.597168 10386 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:40:41.597291 10386 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:40:41.597445 10386 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:40:41.597545 10386 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:40:41.630534 10386 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.3:41985
I20260620 18:40:41.630620 10512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.3:41985 every 8 connection(s)
I20260620 18:40:41.631803 10386 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980527431848-22676-0/minicluster-data/ts-2/data/info.pb
I20260620 18:40:41.641489 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 10386
I20260620 18:40:41.641671 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980527431848-22676-0/minicluster-data/ts-2/wal/instance
I20260620 18:40:41.642253 22676 external_mini_cluster.cc:1428] Setting key 0b2b8c8d914d4b74d2479ca111503fab
I20260620 18:40:41.644976 10254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:41.634480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.3:56649 (local address 127.22.37.62:35347)
0620 18:40:41.634654 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:41.634658 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:41.636031 (+ 1373us) server_negotiation.cc:407] Connection header received
0620 18:40:41.637854 (+ 1823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:41.637860 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:41.637955 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:41.638081 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:40:41.639113 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:41.639861 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:41.641654 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:41.642311 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:41.642437 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:41.644057 (+ 1620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:40:41.644063 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:40:41.644067 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:40:41.644092 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:40:41.644127 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:40:41.644141 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:40:41.644245 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:41.644492 (+ 247us) server_negotiation.cc:299] Negotiation successful
0620 18:40:41.644597 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 18:40:41.646173 10515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:41.634997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:35347 (local address 127.22.37.3:56649)
0620 18:40:41.635873 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:41.635908 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 18:40:41.637592 (+ 1684us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:40:41.638308 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:40:41.638316 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:40:41.638404 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:40:41.638940 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:41.638955 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:41.640031 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:41.640037 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:41.641252 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:41.641263 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:41.642563 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:41.642570 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:41.642782 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:41.643669 (+ 887us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:40:41.643675 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:40:41.643722 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:40:41.643821 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:40:41.644245 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:40:41.644258 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:40:41.644269 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 18:40:41.644477 (+ 208us) client_negotiation.cc:240] Negotiation successful
0620 18:40:41.644876 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":266,"threads_started":1}
I20260620 18:40:41.647500 10513 heartbeater.cc:347] Connected to a master server at 127.22.37.62:35347
I20260620 18:40:41.647800 10513 heartbeater.cc:464] Registering TS with master...
I20260620 18:40:41.648474 10513 heartbeater.cc:511] Master 127.22.37.62:35347 requested a full tablet report, sending...
I20260620 18:40:41.649873 10066 ts_manager.cc:194] Registered new tserver with Master: ce510824a42a4c0eb11b9ddb0b1937bb (127.22.37.3:41985)
I20260620 18:40:41.650597 10066 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.37.3:56649
I20260620 18:40:41.657085 22676 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:40:41.666615 10254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:41.661033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37912 (local address 127.22.37.62:35347)
0620 18:40:41.661232 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:41.661237 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:41.661732 (+ 495us) server_negotiation.cc:407] Connection header received
0620 18:40:41.661856 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:41.661859 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:41.661929 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:41.662006 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:40:41.662457 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:41.663133 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:41.664369 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:41.664745 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:41.664831 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:41.665887 (+ 1056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:40:41.665894 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:40:41.665900 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:40:41.665922 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:40:41.665951 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:40:41.665962 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:40:41.666057 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:41.666217 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 18:40:41.666332 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 18:40:41.670142 10066 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37912:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:40:41.677492 10066 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:40:41.714301 10525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:41.689588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.2:38691 (local address 127.0.0.1:57566)
0620 18:40:41.704124 (+ 14536us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:41.704141 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:40:41.704274 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:40:41.705104 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:40:41.705112 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:40:41.705148 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:40:41.705693 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:41.705709 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:41.707685 (+ 1976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:41.707691 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:41.710018 (+ 2327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:41.710040 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:41.712953 (+ 2913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:41.712963 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:41.713276 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:41.713349 (+ 73us) client_negotiation.cc:769] Sending connection context
0620 18:40:41.713521 (+ 172us) client_negotiation.cc:240] Negotiation successful
0620 18:40:41.713771 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14402,"mutex_wait_us":5,"thread_start_us":14229,"threads_started":1}
I20260620 18:40:41.715723 10526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:41.689403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.3:41985 (local address 127.0.0.1:36908)
0620 18:40:41.702731 (+ 13328us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:41.702793 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 18:40:41.703079 (+ 286us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:40:41.706152 (+ 3073us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:40:41.706157 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:40:41.706189 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:40:41.706634 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:41.706653 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:41.709115 (+ 2462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:41.709123 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:41.712100 (+ 2977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:41.712121 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:41.714567 (+ 2446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:41.714573 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:41.714834 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:41.714888 (+ 54us) client_negotiation.cc:769] Sending connection context
0620 18:40:41.714979 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 18:40:41.715169 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12939,"mutex_wait_us":51,"thread_start_us":15879,"threads_started":1}
I20260620 18:40:41.715951 10527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:41.689837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57566 (local address 127.22.37.2:38691)
0620 18:40:41.703758 (+ 13921us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:41.703770 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:41.704429 (+ 659us) server_negotiation.cc:407] Connection header received
0620 18:40:41.704541 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:41.704552 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:41.704831 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:41.705051 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:40:41.705928 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:41.707516 (+ 1588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:41.710305 (+ 2789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:41.712274 (+ 1969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:41.712438 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:41.712788 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:41.715166 (+ 2378us) server_negotiation.cc:299] Negotiation successful
0620 18:40:41.715413 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13757,"thread_start_us":13661,"threads_started":1}
I20260620 18:40:41.717635 10528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:41.689739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36908 (local address 127.22.37.3:41985)
0620 18:40:41.705112 (+ 15373us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:41.705123 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:41.705161 (+ 38us) server_negotiation.cc:407] Connection header received
0620 18:40:41.705279 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:41.705288 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:41.705650 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:41.705835 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:40:41.706879 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:41.708926 (+ 2047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:41.712384 (+ 3458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:41.714334 (+ 1950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:41.716569 (+ 2235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:41.716828 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:41.716954 (+ 126us) server_negotiation.cc:299] Negotiation successful
0620 18:40:41.717121 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15058,"thread_start_us":14898,"threads_started":1}
I20260620 18:40:41.720557 10318 tablet_service.cc:1467] Processing CreateTablet for tablet 5c78c76f7d2c4ac8adc808918dfd6a17 (DEFAULT_TABLE table=test-table [id=8dc2d158d69a4172ad9ee51386931b8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:40:41.722093 10318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c78c76f7d2c4ac8adc808918dfd6a17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:40:41.723006 10448 tablet_service.cc:1467] Processing CreateTablet for tablet 5c78c76f7d2c4ac8adc808918dfd6a17 (DEFAULT_TABLE table=test-table [id=8dc2d158d69a4172ad9ee51386931b8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:40:41.724148 10448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c78c76f7d2c4ac8adc808918dfd6a17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:40:41.725085 10525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:41.709800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.1:36117 (local address 127.0.0.1:42476)
0620 18:40:41.714809 (+ 5009us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:41.714833 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 18:40:41.714958 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:40:41.719003 (+ 4045us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:40:41.719008 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:40:41.719032 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:40:41.719210 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:41.719224 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:41.720713 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:41.720719 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:41.722488 (+ 1769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:41.722506 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:41.724297 (+ 1791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:41.724303 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:41.724480 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:41.724524 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 18:40:41.724623 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 18:40:41.724770 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4869,"thread_start_us":8555,"threads_started":1}
I20260620 18:40:41.725221 10529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:41.706079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42476 (local address 127.22.37.1:36117)
0620 18:40:41.718329 (+ 12250us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:41.718339 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:41.718371 (+ 32us) server_negotiation.cc:407] Connection header received
0620 18:40:41.718482 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:41.718491 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:41.718758 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:41.719006 (+ 248us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:40:41.719369 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:41.720543 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:41.722684 (+ 2141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:41.724109 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:41.724249 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:41.724504 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:41.724692 (+ 188us) server_negotiation.cc:299] Negotiation successful
0620 18:40:41.724859 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12063,"thread_start_us":11956,"threads_started":1}
I20260620 18:40:41.729449 10187 tablet_service.cc:1467] Processing CreateTablet for tablet 5c78c76f7d2c4ac8adc808918dfd6a17 (DEFAULT_TABLE table=test-table [id=8dc2d158d69a4172ad9ee51386931b8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:40:41.730403 10252 heartbeater.cc:503] Master 127.22.37.62:35347 was elected leader, sending a full tablet report...
I20260620 18:40:41.730880 10187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c78c76f7d2c4ac8adc808918dfd6a17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:40:41.735634 10531 tablet_bootstrap.cc:492] T 5c78c76f7d2c4ac8adc808918dfd6a17 P abbe673979ae4b509c91573c23599e36: Bootstrap starting.
I20260620 18:40:41.737749 10532 tablet_bootstrap.cc:492] T 5c78c76f7d2c4ac8adc808918dfd6a17 P ce510824a42a4c0eb11b9ddb0b1937bb: Bootstrap starting.
I20260620 18:40:41.739217 10531 tablet_bootstrap.cc:654] T 5c78c76f7d2c4ac8adc808918dfd6a17 P abbe673979ae4b509c91573c23599e36: Neither blocks nor log segments found. Creating new log.
I20260620 18:40:41.740202 10531 log.cc:824] T 5c78c76f7d2c4ac8adc808918dfd6a17 P abbe673979ae4b509c91573c23599e36: Log is configured to *not* fsync() on all Append() calls
I20260620 18:40:41.741174 10532 tablet_bootstrap.cc:654] T 5c78c76f7d2c4ac8adc808918dfd6a17 P ce510824a42a4c0eb11b9ddb0b1937bb: Neither blocks nor log segments found. Creating new log.
I20260620 18:40:41.742434 10532 log.cc:824] T 5c78c76f7d2c4ac8adc808918dfd6a17 P ce510824a42a4c0eb11b9ddb0b1937bb: Log is configured to *not* fsync() on all Append() calls
I20260620 18:40:41.743388 10531 tablet_bootstrap.cc:492] T 5c78c76f7d2c4ac8adc808918dfd6a17 P abbe673979ae4b509c91573c23599e36: No bootstrap required, opened a new log
I20260620 18:40:41.743686 10531 ts_tablet_manager.cc:1397] T 5c78c76f7d2c4ac8adc808918dfd6a17 P abbe673979ae4b509c91573c23599e36: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.007s
I20260620 18:40:41.745546 10532 tablet_bootstrap.cc:492] T 5c78c76f7d2c4ac8adc808918dfd6a17 P ce510824a42a4c0eb11b9ddb0b1937bb: No bootstrap required, opened a new log
I20260620 18:40:41.745803 10532 ts_tablet_manager.cc:1397] T 5c78c76f7d2c4ac8adc808918dfd6a17 P ce510824a42a4c0eb11b9ddb0b1937bb: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260620 18:40:41.748492 10535 tablet_bootstrap.cc:492] T 5c78c76f7d2c4ac8adc808918dfd6a17 P 3abf467f21b74971a4b252a8fe60627a: Bootstrap starting.
I20260620 18:40:41.752090 10535 tablet_bootstrap.cc:654] T 5c78c76f7d2c4ac8adc808918dfd6a17 P 3abf467f21b74971a4b252a8fe60627a: Neither blocks nor log segments found. Creating new log.
I20260620 18:40:41.752840 10531 raft_consensus.cc:348] T 5c78c76f7d2c4ac8adc808918dfd6a17 P abbe673979ae4b509c91573c23599e36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce510824a42a4c0eb11b9ddb0b1937bb" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 41985 } } peers { permanent_uuid: "abbe673979ae4b509c91573c23599e36" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 38691 } } peers { permanent_uuid: "3abf467f21b74971a4b252a8fe60627a" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 36117 } }
I20260620 18:40:41.753305 10531 raft_consensus.cc:374] T 5c78c76f7d2c4ac8adc808918dfd6a17 P abbe673979ae4b509c91573c23599e36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:40:41.753361 10535 log.cc:824] T 5c78c76f7d2c4ac8adc808918dfd6a17 P 3abf467f21b74971a4b252a8fe60627a: Log is configured to *not* fsync() on all Append() calls
I20260620 18:40:41.753456 10531 raft_consensus.cc:729] T 5c78c76f7d2c4ac8adc808918dfd6a17 P abbe673979ae4b509c91573c23599e36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abbe673979ae4b509c91573c23599e36, State: Initialized, Role: FOLLOWER
I20260620 18:40:41.754122 10531 consensus_queue.cc:260] T 5c78c76f7d2c4ac8adc808918dfd6a17 P abbe673979ae4b509c91573c23599e36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce510824a42a4c0eb11b9ddb0b1937bb" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 41985 } } peers { permanent_uuid: "abbe673979ae4b509c91573c23599e36" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 38691 } } peers { permanent_uuid: "3abf467f21b74971a4b252a8fe60627a" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 36117 } }
I20260620 18:40:41.754930 10532 raft_consensus.cc:348] T 5c78c76f7d2c4ac8adc808918dfd6a17 P ce510824a42a4c0eb11b9ddb0b1937bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce510824a42a4c0eb11b9ddb0b1937bb" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 41985 } } peers { permanent_uuid: "abbe673979ae4b509c91573c23599e36" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 38691 } } peers { permanent_uuid: "3abf467f21b74971a4b252a8fe60627a" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 36117 } }
I20260620 18:40:41.755267 10532 raft_consensus.cc:374] T 5c78c76f7d2c4ac8adc808918dfd6a17 P ce510824a42a4c0eb11b9ddb0b1937bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:40:41.755389 10532 raft_consensus.cc:729] T 5c78c76f7d2c4ac8adc808918dfd6a17 P ce510824a42a4c0eb11b9ddb0b1937bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce510824a42a4c0eb11b9ddb0b1937bb, State: Initialized, Role: FOLLOWER
I20260620 18:40:41.755942 10532 consensus_queue.cc:260] T 5c78c76f7d2c4ac8adc808918dfd6a17 P ce510824a42a4c0eb11b9ddb0b1937bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce510824a42a4c0eb11b9ddb0b1937bb" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 41985 } } peers { permanent_uuid: "abbe673979ae4b509c91573c23599e36" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 38691 } } peers { permanent_uuid: "3abf467f21b74971a4b252a8fe60627a" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 36117 } }
I20260620 18:40:41.756752 10535 tablet_bootstrap.cc:492] T 5c78c76f7d2c4ac8adc808918dfd6a17 P 3abf467f21b74971a4b252a8fe60627a: No bootstrap required, opened a new log
I20260620 18:40:41.757066 10535 ts_tablet_manager.cc:1397] T 5c78c76f7d2c4ac8adc808918dfd6a17 P 3abf467f21b74971a4b252a8fe60627a: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20260620 18:40:41.757531 10532 ts_tablet_manager.cc:1428] T 5c78c76f7d2c4ac8adc808918dfd6a17 P ce510824a42a4c0eb11b9ddb0b1937bb: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260620 18:40:41.761626 10513 heartbeater.cc:503] Master 127.22.37.62:35347 was elected leader, sending a full tablet report...
I20260620 18:40:41.762502 10383 heartbeater.cc:503] Master 127.22.37.62:35347 was elected leader, sending a full tablet report...
I20260620 18:40:41.762706 10531 ts_tablet_manager.cc:1428] T 5c78c76f7d2c4ac8adc808918dfd6a17 P abbe673979ae4b509c91573c23599e36: Time spent starting tablet: real 0.019s user 0.017s sys 0.001s
I20260620 18:40:41.769461 10535 raft_consensus.cc:348] T 5c78c76f7d2c4ac8adc808918dfd6a17 P 3abf467f21b74971a4b252a8fe60627a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce510824a42a4c0eb11b9ddb0b1937bb" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 41985 } } peers { permanent_uuid: "abbe673979ae4b509c91573c23599e36" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 38691 } } peers { permanent_uuid: "3abf467f21b74971a4b252a8fe60627a" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 36117 } }
I20260620 18:40:41.769881 10535 raft_consensus.cc:374] T 5c78c76f7d2c4ac8adc808918dfd6a17 P 3abf467f21b74971a4b252a8fe60627a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:40:41.769981 10535 raft_consensus.cc:729] T 5c78c76f7d2c4ac8adc808918dfd6a17 P 3abf467f21b74971a4b252a8fe60627a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3abf467f21b74971a4b252a8fe60627a, State: Initialized, Role: FOLLOWER
I20260620 18:40:41.770658 10535 consensus_queue.cc:260] T 5c78c76f7d2c4ac8adc808918dfd6a17 P 3abf467f21b74971a4b252a8fe60627a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce510824a42a4c0eb11b9ddb0b1937bb" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 41985 } } peers { permanent_uuid: "abbe673979ae4b509c91573c23599e36" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 38691 } } peers { permanent_uuid: "3abf467f21b74971a4b252a8fe60627a" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 36117 } }
I20260620 18:40:41.772467 10535 ts_tablet_manager.cc:1428] T 5c78c76f7d2c4ac8adc808918dfd6a17 P 3abf467f21b74971a4b252a8fe60627a: Time spent starting tablet: real 0.015s user 0.010s sys 0.003s
W20260620 18:40:41.883503 10514 tablet.cc:2369] T 5c78c76f7d2c4ac8adc808918dfd6a17 P ce510824a42a4c0eb11b9ddb0b1937bb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:40:41.925938 10384 tablet.cc:2369] T 5c78c76f7d2c4ac8adc808918dfd6a17 P abbe673979ae4b509c91573c23599e36: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:40:41.955324 10253 tablet.cc:2369] T 5c78c76f7d2c4ac8adc808918dfd6a17 P 3abf467f21b74971a4b252a8fe60627a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:40:41.960768 10538 raft_consensus.cc:482] T 5c78c76f7d2c4ac8adc808918dfd6a17 P ce510824a42a4c0eb11b9ddb0b1937bb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:40:41.961082 10538 raft_consensus.cc:504] T 5c78c76f7d2c4ac8adc808918dfd6a17 P ce510824a42a4c0eb11b9ddb0b1937bb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce510824a42a4c0eb11b9ddb0b1937bb" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 41985 } } peers { permanent_uuid: "abbe673979ae4b509c91573c23599e36" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 38691 } } peers { permanent_uuid: "3abf467f21b74971a4b252a8fe60627a" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 36117 } }
I20260620 18:40:41.962848 10538 leader_election.cc:288] T 5c78c76f7d2c4ac8adc808918dfd6a17 P ce510824a42a4c0eb11b9ddb0b1937bb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers abbe673979ae4b509c91573c23599e36 (127.22.37.2:38691), 3abf467f21b74971a4b252a8fe60627a (127.22.37.1:36117)
I20260620 18:40:41.963765 10537 raft_consensus.cc:482] T 5c78c76f7d2c4ac8adc808918dfd6a17 P abbe673979ae4b509c91573c23599e36 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:40:41.964088 10537 raft_consensus.cc:504] T 5c78c76f7d2c4ac8adc808918dfd6a17 P abbe673979ae4b509c91573c23599e36 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce510824a42a4c0eb11b9ddb0b1937bb" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 41985 } } peers { permanent_uuid: "abbe673979ae4b509c91573c23599e36" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 38691 } } peers { permanent_uuid: "3abf467f21b74971a4b252a8fe60627a" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 36117 } }
I20260620 18:40:41.966089 10537 leader_election.cc:288] T 5c78c76f7d2c4ac8adc808918dfd6a17 P abbe673979ae4b509c91573c23599e36 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce510824a42a4c0eb11b9ddb0b1937bb (127.22.37.3:41985), 3abf467f21b74971a4b252a8fe60627a (127.22.37.1:36117)
I20260620 18:40:41.970827 10515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:41.963282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.1:36117 (local address 127.22.37.3:55915)
0620 18:40:41.963593 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:41.963616 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 18:40:41.963775 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:40:41.964401 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:40:41.964407 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:40:41.964436 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:40:41.964649 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:41.964658 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:41.966053 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:41.966060 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:41.968272 (+ 2212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:41.968285 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:41.970020 (+ 1735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:41.970026 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:41.970216 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:41.970271 (+ 55us) client_negotiation.cc:769] Sending connection context
0620 18:40:41.970368 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 18:40:41.970509 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"mutex_wait_us":1}
I20260620 18:40:41.971263 10529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:41.963354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.3:55915 (local address 127.22.37.1:36117)
0620 18:40:41.963561 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:41.963568 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:41.963705 (+ 137us) server_negotiation.cc:407] Connection header received
0620 18:40:41.964115 (+ 410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:41.964122 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:41.964224 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:41.964347 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:40:41.964836 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:41.965881 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:41.968458 (+ 2577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:41.969713 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:41.969833 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:41.969931 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:41.970704 (+ 773us) server_negotiation.cc:299] Negotiation successful
0620 18:40:41.970820 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 18:40:41.972291 10207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c78c76f7d2c4ac8adc808918dfd6a17" candidate_uuid: "ce510824a42a4c0eb11b9ddb0b1937bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3abf467f21b74971a4b252a8fe60627a" is_pre_election: true
I20260620 18:40:41.972841 10527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:41.963363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.3:49519 (local address 127.22.37.2:38691)
0620 18:40:41.963567 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:41.963572 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:41.964552 (+ 980us) server_negotiation.cc:407] Connection header received
0620 18:40:41.964764 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:41.964771 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:41.964870 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:41.964967 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:40:41.965541 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:41.966479 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:41.969485 (+ 3006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:41.971366 (+ 1881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:41.971517 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:41.971656 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:41.972243 (+ 587us) server_negotiation.cc:299] Negotiation successful
0620 18:40:41.972411 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":119,"server-negotiator.queue_time_us":62}
I20260620 18:40:41.972817 10207 raft_consensus.cc:2454] T 5c78c76f7d2c4ac8adc808918dfd6a17 P 3abf467f21b74971a4b252a8fe60627a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce510824a42a4c0eb11b9ddb0b1937bb in term 0.
I20260620 18:40:41.973822 10540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:41.963282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.2:38691 (local address 127.22.37.3:49519)
0620 18:40:41.964373 (+ 1091us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:41.964393 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:40:41.964604 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:40:41.964990 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:40:41.964994 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:40:41.965017 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:40:41.965330 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:41.965337 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:41.966711 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:41.966720 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:41.969220 (+ 2500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:41.969237 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:41.971774 (+ 2537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:41.971783 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:41.972018 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:41.972068 (+ 50us) client_negotiation.cc:769] Sending connection context
0620 18:40:41.973092 (+ 1024us) client_negotiation.cc:240] Negotiation successful
0620 18:40:41.973291 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":961,"mutex_wait_us":237,"thread_start_us":388,"threads_started":1}
I20260620 18:40:41.973942 10402 leader_election.cc:302] T 5c78c76f7d2c4ac8adc808918dfd6a17 P ce510824a42a4c0eb11b9ddb0b1937bb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3abf467f21b74971a4b252a8fe60627a, ce510824a42a4c0eb11b9ddb0b1937bb; no voters:
I20260620 18:40:41.974376 10538 raft_consensus.cc:2781] T 5c78c76f7d2c4ac8adc808918dfd6a17 P ce510824a42a4c0eb11b9ddb0b1937bb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:40:41.974543 10538 raft_consensus.cc:482] T 5c78c76f7d2c4ac8adc808918dfd6a17 P ce510824a42a4c0eb11b9ddb0b1937bb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:40:41.974656 10538 raft_consensus.cc:3037] T 5c78c76f7d2c4ac8adc808918dfd6a17 P ce510824a42a4c0eb11b9ddb0b1937bb [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:40:41.975016 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c78c76f7d2c4ac8adc808918dfd6a17" candidate_uuid: "ce510824a42a4c0eb11b9ddb0b1937bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abbe673979ae4b509c91573c23599e36" is_pre_election: true
I20260620 18:40:41.975463 10338 raft_consensus.cc:2454] T 5c78c76f7d2c4ac8adc808918dfd6a17 P abbe673979ae4b509c91573c23599e36 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce510824a42a4c0eb11b9ddb0b1937bb in term 0.
I20260620 18:40:41.975571 10528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:41.966499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.2:43595 (local address 127.22.37.3:41985)
0620 18:40:41.966660 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:41.966666 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:41.967217 (+ 551us) server_negotiation.cc:407] Connection header received
0620 18:40:41.967433 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:41.967438 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:41.967494 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:41.967548 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:40:41.968292 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:41.969447 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:41.973096 (+ 3649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:41.974375 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:41.974485 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:41.974585 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:41.975161 (+ 576us) server_negotiation.cc:299] Negotiation successful
0620 18:40:41.975294 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 18:40:41.975637 10541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:41.966372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.3:41985 (local address 127.22.37.2:43595)
0620 18:40:41.967121 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:41.967145 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 18:40:41.967307 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:40:41.967702 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:40:41.967706 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:40:41.967733 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:40:41.968132 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:41.968143 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:41.969621 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:41.969627 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:41.972852 (+ 3225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:41.972867 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:41.974754 (+ 1887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:41.974765 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:41.974957 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:41.974998 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 18:40:41.975121 (+ 123us) client_negotiation.cc:240] Negotiation successful
0620 18:40:41.975273 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":436,"threads_started":1}
I20260620 18:40:41.977686 10468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c78c76f7d2c4ac8adc808918dfd6a17" candidate_uuid: "abbe673979ae4b509c91573c23599e36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce510824a42a4c0eb11b9ddb0b1937bb" is_pre_election: true
I20260620 18:40:41.978128 10538 raft_consensus.cc:504] T 5c78c76f7d2c4ac8adc808918dfd6a17 P ce510824a42a4c0eb11b9ddb0b1937bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce510824a42a4c0eb11b9ddb0b1937bb" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 41985 } } peers { permanent_uuid: "abbe673979ae4b509c91573c23599e36" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 38691 } } peers { permanent_uuid: "3abf467f21b74971a4b252a8fe60627a" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 36117 } }
I20260620 18:40:41.978664 10468 raft_consensus.cc:2379] T 5c78c76f7d2c4ac8adc808918dfd6a17 P ce510824a42a4c0eb11b9ddb0b1937bb [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate abbe673979ae4b509c91573c23599e36 in current term 1: Already voted for candidate ce510824a42a4c0eb11b9ddb0b1937bb in this term.
I20260620 18:40:41.978948 10538 leader_election.cc:288] T 5c78c76f7d2c4ac8adc808918dfd6a17 P ce510824a42a4c0eb11b9ddb0b1937bb [CANDIDATE]: Term 1 election: Requested vote from peers abbe673979ae4b509c91573c23599e36 (127.22.37.2:38691), 3abf467f21b74971a4b252a8fe60627a (127.22.37.1:36117)
I20260620 18:40:41.979235 10207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c78c76f7d2c4ac8adc808918dfd6a17" candidate_uuid: "ce510824a42a4c0eb11b9ddb0b1937bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3abf467f21b74971a4b252a8fe60627a"
I20260620 18:40:41.979456 10207 raft_consensus.cc:3037] T 5c78c76f7d2c4ac8adc808918dfd6a17 P 3abf467f21b74971a4b252a8fe60627a [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:40:41.979594 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c78c76f7d2c4ac8adc808918dfd6a17" candidate_uuid: "ce510824a42a4c0eb11b9ddb0b1937bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abbe673979ae4b509c91573c23599e36"
I20260620 18:40:41.979795 10338 raft_consensus.cc:3037] T 5c78c76f7d2c4ac8adc808918dfd6a17 P abbe673979ae4b509c91573c23599e36 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:40:41.982939 10207 raft_consensus.cc:2454] T 5c78c76f7d2c4ac8adc808918dfd6a17 P 3abf467f21b74971a4b252a8fe60627a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce510824a42a4c0eb11b9ddb0b1937bb in term 1.
I20260620 18:40:41.982990 10338 raft_consensus.cc:2454] T 5c78c76f7d2c4ac8adc808918dfd6a17 P abbe673979ae4b509c91573c23599e36 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce510824a42a4c0eb11b9ddb0b1937bb in term 1.
I20260620 18:40:41.983364 10403 leader_election.cc:302] T 5c78c76f7d2c4ac8adc808918dfd6a17 P ce510824a42a4c0eb11b9ddb0b1937bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: abbe673979ae4b509c91573c23599e36, ce510824a42a4c0eb11b9ddb0b1937bb; no voters:
I20260620 18:40:41.983713 10538 raft_consensus.cc:2781] T 5c78c76f7d2c4ac8adc808918dfd6a17 P ce510824a42a4c0eb11b9ddb0b1937bb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:40:41.984241 10538 raft_consensus.cc:686] T 5c78c76f7d2c4ac8adc808918dfd6a17 P ce510824a42a4c0eb11b9ddb0b1937bb [term 1 LEADER]: Becoming Leader. State: Replica: ce510824a42a4c0eb11b9ddb0b1937bb, State: Running, Role: LEADER
I20260620 18:40:41.984418 10541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:41.967223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.1:36117 (local address 127.22.37.2:44085)
0620 18:40:41.975842 (+ 8619us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:41.975863 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 18:40:41.975957 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:40:41.976349 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:40:41.976354 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:40:41.976383 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:40:41.976532 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:41.976541 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:41.978126 (+ 1585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:41.978131 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:41.982039 (+ 3908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:41.982064 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:41.983537 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:41.983543 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:41.983733 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:41.983758 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 18:40:41.983974 (+ 216us) client_negotiation.cc:240] Negotiation successful
0620 18:40:41.984128 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8519,"thread_start_us":9485,"threads_started":1}
I20260620 18:40:41.984539 10543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:41.967695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.37.2:44085 (local address 127.22.37.1:36117)
0620 18:40:41.970626 (+ 2931us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:41.970631 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:41.976071 (+ 5440us) server_negotiation.cc:407] Connection header received
0620 18:40:41.976131 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:41.976136 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:41.976210 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:41.976309 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:40:41.976748 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:41.977993 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:41.982218 (+ 4225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:41.983393 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:41.983502 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:41.983636 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:41.984000 (+ 364us) server_negotiation.cc:299] Negotiation successful
0620 18:40:41.984132 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2815,"thread_start_us":2658,"threads_started":1}
I20260620 18:40:41.985111 10207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c78c76f7d2c4ac8adc808918dfd6a17" candidate_uuid: "abbe673979ae4b509c91573c23599e36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3abf467f21b74971a4b252a8fe60627a" is_pre_election: true
I20260620 18:40:41.985138 10538 consensus_queue.cc:237] T 5c78c76f7d2c4ac8adc808918dfd6a17 P ce510824a42a4c0eb11b9ddb0b1937bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce510824a42a4c0eb11b9ddb0b1937bb" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 41985 } } peers { permanent_uuid: "abbe673979ae4b509c91573c23599e36" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 38691 } } peers { permanent_uuid: "3abf467f21b74971a4b252a8fe60627a" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 36117 } }
I20260620 18:40:41.985461 10207 raft_consensus.cc:2379] T 5c78c76f7d2c4ac8adc808918dfd6a17 P 3abf467f21b74971a4b252a8fe60627a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate abbe673979ae4b509c91573c23599e36 in current term 1: Already voted for candidate ce510824a42a4c0eb11b9ddb0b1937bb in this term.
I20260620 18:40:41.986069 10272 leader_election.cc:302] T 5c78c76f7d2c4ac8adc808918dfd6a17 P abbe673979ae4b509c91573c23599e36 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: abbe673979ae4b509c91573c23599e36; no voters: 3abf467f21b74971a4b252a8fe60627a, ce510824a42a4c0eb11b9ddb0b1937bb
I20260620 18:40:41.986479 10537 raft_consensus.cc:2726] T 5c78c76f7d2c4ac8adc808918dfd6a17 P abbe673979ae4b509c91573c23599e36 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 18:40:41.989882 10065 catalog_manager.cc:5495] T 5c78c76f7d2c4ac8adc808918dfd6a17 P ce510824a42a4c0eb11b9ddb0b1937bb reported cstate change: term changed from 0 to 1, leader changed from <none> to ce510824a42a4c0eb11b9ddb0b1937bb (127.22.37.3). New cstate: current_term: 1 leader_uuid: "ce510824a42a4c0eb11b9ddb0b1937bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce510824a42a4c0eb11b9ddb0b1937bb" member_type: VOTER last_known_addr { host: "127.22.37.3" port: 41985 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "abbe673979ae4b509c91573c23599e36" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 38691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3abf467f21b74971a4b252a8fe60627a" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 36117 } health_report { overall_health: UNKNOWN } } }
I20260620 18:40:42.028029 10528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:42.022369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36920 (local address 127.22.37.3:41985)
0620 18:40:42.022579 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:42.022585 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:42.022694 (+ 109us) server_negotiation.cc:407] Connection header received
0620 18:40:42.022949 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:42.022953 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:42.023064 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:42.023214 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:40:42.023587 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:42.024337 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:42.025825 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:42.026253 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:42.026383 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:42.026898 (+ 515us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:40:42.027425 (+ 527us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:40:42.027609 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:42.027658 (+ 49us) server_negotiation.cc:299] Negotiation successful
0620 18:40:42.027739 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 18:40:42.036403 10207 raft_consensus.cc:1261] T 5c78c76f7d2c4ac8adc808918dfd6a17 P 3abf467f21b74971a4b252a8fe60627a [term 1 FOLLOWER]: Refusing update from remote peer ce510824a42a4c0eb11b9ddb0b1937bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:40:42.036446 10338 raft_consensus.cc:1261] T 5c78c76f7d2c4ac8adc808918dfd6a17 P abbe673979ae4b509c91573c23599e36 [term 1 FOLLOWER]: Refusing update from remote peer ce510824a42a4c0eb11b9ddb0b1937bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:40:42.037509 10538 consensus_queue.cc:1035] T 5c78c76f7d2c4ac8adc808918dfd6a17 P ce510824a42a4c0eb11b9ddb0b1937bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3abf467f21b74971a4b252a8fe60627a" member_type: VOTER last_known_addr { host: "127.22.37.1" port: 36117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:40:42.037843 10544 consensus_queue.cc:1035] T 5c78c76f7d2c4ac8adc808918dfd6a17 P ce510824a42a4c0eb11b9ddb0b1937bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "abbe673979ae4b509c91573c23599e36" member_type: VOTER last_known_addr { host: "127.22.37.2" port: 38691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:40:42.051465 10547 mvcc.cc:205] Tried to move back new op lower bound from 7298993528971022336 to 7298993528775909376. Current Snapshot: MvccSnapshot[applied={T|T < 7298993528971022336}]
I20260620 18:40:42.064239 10066 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37912:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:40:42.064574 10066 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37912:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:40:42.068491 10066 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5a2ccdf5aa55489dbba20dcaacb2b3c4: Sending DeleteTablet for 3 replicas of tablet 5c78c76f7d2c4ac8adc808918dfd6a17
I20260620 18:40:42.069834 10448 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c78c76f7d2c4ac8adc808918dfd6a17 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:40:42 UTC) from {username='slave'} at 127.0.0.1:36908
I20260620 18:40:42.070072 10187 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c78c76f7d2c4ac8adc808918dfd6a17 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:40:42 UTC) from {username='slave'} at 127.0.0.1:42476
I20260620 18:40:42.070538 10318 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c78c76f7d2c4ac8adc808918dfd6a17 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:40:42 UTC) from {username='slave'} at 127.0.0.1:57566
I20260620 18:40:42.071189 10558 tablet_replica.cc:330] T 5c78c76f7d2c4ac8adc808918dfd6a17 P 3abf467f21b74971a4b252a8fe60627a: stopping tablet replica
I20260620 18:40:42.071385 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 10125
I20260620 18:40:42.071717 10558 raft_consensus.cc:2229] T 5c78c76f7d2c4ac8adc808918dfd6a17 P 3abf467f21b74971a4b252a8fe60627a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:40:42.072391 10558 raft_consensus.cc:2258] T 5c78c76f7d2c4ac8adc808918dfd6a17 P 3abf467f21b74971a4b252a8fe60627a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:40:42.075747 10558 ts_tablet_manager.cc:1905] T 5c78c76f7d2c4ac8adc808918dfd6a17 P 3abf467f21b74971a4b252a8fe60627a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:40:42.082639 10543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:42.072615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42484 (local address 127.22.37.1:36117)
0620 18:40:42.072805 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:42.072812 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:42.073901 (+ 1089us) server_negotiation.cc:407] Connection header received
0620 18:40:42.074202 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:42.074208 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:42.074330 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:42.074427 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:40:42.075190 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:42.076194 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:42.078638 (+ 2444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:42.079260 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:42.079370 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:42.080915 (+ 1545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:40:42.080926 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:40:42.080949 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:40:42.081787 (+ 838us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:40:42.081852 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:40:42.081882 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:40:42.081968 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:42.082194 (+ 226us) server_negotiation.cc:299] Negotiation successful
0620 18:40:42.082292 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 18:40:42.083089 10558 ts_tablet_manager.cc:1918] T 5c78c76f7d2c4ac8adc808918dfd6a17 P 3abf467f21b74971a4b252a8fe60627a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:40:42.083292 10558 log.cc:1196] T 5c78c76f7d2c4ac8adc808918dfd6a17 P 3abf467f21b74971a4b252a8fe60627a: Deleting WAL directory at /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980527431848-22676-0/minicluster-data/ts-0/wal/wals/5c78c76f7d2c4ac8adc808918dfd6a17
I20260620 18:40:42.084059 10559 tablet_replica.cc:330] T 5c78c76f7d2c4ac8adc808918dfd6a17 P abbe673979ae4b509c91573c23599e36: stopping tablet replica
I20260620 18:40:42.084182 10247 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:40:42.084847 10559 raft_consensus.cc:2229] T 5c78c76f7d2c4ac8adc808918dfd6a17 P abbe673979ae4b509c91573c23599e36 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:40:42.085587 10559 raft_consensus.cc:2258] T 5c78c76f7d2c4ac8adc808918dfd6a17 P abbe673979ae4b509c91573c23599e36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:40:42.086608 10557 tablet_replica.cc:330] T 5c78c76f7d2c4ac8adc808918dfd6a17 P ce510824a42a4c0eb11b9ddb0b1937bb: stopping tablet replica
I20260620 18:40:42.087167 10557 raft_consensus.cc:2229] T 5c78c76f7d2c4ac8adc808918dfd6a17 P ce510824a42a4c0eb11b9ddb0b1937bb [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:40:42.088091 10559 ts_tablet_manager.cc:1905] T 5c78c76f7d2c4ac8adc808918dfd6a17 P abbe673979ae4b509c91573c23599e36: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:40:42.088115 10557 raft_consensus.cc:2258] T 5c78c76f7d2c4ac8adc808918dfd6a17 P ce510824a42a4c0eb11b9ddb0b1937bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:40:42.090404 10557 ts_tablet_manager.cc:1905] T 5c78c76f7d2c4ac8adc808918dfd6a17 P ce510824a42a4c0eb11b9ddb0b1937bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:40:42.097604 10557 ts_tablet_manager.cc:1918] T 5c78c76f7d2c4ac8adc808918dfd6a17 P ce510824a42a4c0eb11b9ddb0b1937bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:40:42.097776 10557 log.cc:1196] T 5c78c76f7d2c4ac8adc808918dfd6a17 P ce510824a42a4c0eb11b9ddb0b1937bb: Deleting WAL directory at /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980527431848-22676-0/minicluster-data/ts-2/wal/wals/5c78c76f7d2c4ac8adc808918dfd6a17
I20260620 18:40:42.098342 10557 ts_tablet_manager.cc:1939] T 5c78c76f7d2c4ac8adc808918dfd6a17 P ce510824a42a4c0eb11b9ddb0b1937bb: Deleting consensus metadata
I20260620 18:40:42.098860 10559 ts_tablet_manager.cc:1918] T 5c78c76f7d2c4ac8adc808918dfd6a17 P abbe673979ae4b509c91573c23599e36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:40:42.099080 10559 log.cc:1196] T 5c78c76f7d2c4ac8adc808918dfd6a17 P abbe673979ae4b509c91573c23599e36: Deleting WAL directory at /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980527431848-22676-0/minicluster-data/ts-1/wal/wals/5c78c76f7d2c4ac8adc808918dfd6a17
I20260620 18:40:42.099759 10559 ts_tablet_manager.cc:1939] T 5c78c76f7d2c4ac8adc808918dfd6a17 P abbe673979ae4b509c91573c23599e36: Deleting consensus metadata
I20260620 18:40:42.101307 10051 catalog_manager.cc:4841] TS ce510824a42a4c0eb11b9ddb0b1937bb (127.22.37.3:41985): tablet 5c78c76f7d2c4ac8adc808918dfd6a17 (table test-table [id=8dc2d158d69a4172ad9ee51386931b8e]) successfully deleted
I20260620 18:40:42.101902 10052 catalog_manager.cc:4841] TS abbe673979ae4b509c91573c23599e36 (127.22.37.2:38691): tablet 5c78c76f7d2c4ac8adc808918dfd6a17 (table test-table [id=8dc2d158d69a4172ad9ee51386931b8e]) successfully deleted
I20260620 18:40:42.217998 10558 ts_tablet_manager.cc:1939] T 5c78c76f7d2c4ac8adc808918dfd6a17 P 3abf467f21b74971a4b252a8fe60627a: Deleting consensus metadata
I20260620 18:40:42.218748 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 10125
I20260620 18:40:42.236625 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 10256
W20260620 18:40:42.236663 10051 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.37.1:36117 (error 108)
I20260620 18:40:42.238895 10563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:42.238108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.1:36117 (local address 127.0.0.1:42488)
0620 18:40:42.238549 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:42.238676 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.1:36117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":207,"threads_started":1}
W20260620 18:40:42.239229 10051 catalog_manager.cc:4584] TS 3abf467f21b74971a4b252a8fe60627a (127.22.37.1:36117): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5c78c76f7d2c4ac8adc808918dfd6a17: Network error: Client connection negotiation failed: client connection to 127.22.37.1:36117: connect: Connection refused (error 111)
I20260620 18:40:42.244755 10527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:42.237632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57574 (local address 127.22.37.2:38691)
0620 18:40:42.237824 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:42.237830 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:42.237850 (+ 20us) server_negotiation.cc:407] Connection header received
0620 18:40:42.238100 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:42.238104 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:42.238179 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:42.238297 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:40:42.238900 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:42.240009 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:42.241268 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:42.241948 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:42.242077 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:42.243084 (+ 1007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:40:42.243095 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:40:42.243114 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:40:42.243934 (+ 820us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:40:42.243995 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:40:42.244021 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:40:42.244133 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:42.244340 (+ 207us) server_negotiation.cc:299] Negotiation successful
0620 18:40:42.244441 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 18:40:42.245460 10378 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:40:42.292857 10563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:42.292301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.1:36117 (local address 127.0.0.1:42490)
0620 18:40:42.292552 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:42.292657 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.1:36117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260620 18:40:42.294077 10563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:42.293619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.1:36117 (local address 127.0.0.1:42502)
0620 18:40:42.293795 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:42.293887 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.1:36117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260620 18:40:42.352794 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 10256
I20260620 18:40:42.354933 10563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:42.354309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.1:36117 (local address 127.0.0.1:42516)
0620 18:40:42.354634 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:42.354737 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.1:36117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191}
I20260620 18:40:42.356256 10563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:42.355766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.1:36117 (local address 127.0.0.1:42522)
0620 18:40:42.355973 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:42.356061 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.1:36117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260620 18:40:42.371161 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 10386
I20260620 18:40:42.379299 10528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:42.371942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36930 (local address 127.22.37.3:41985)
0620 18:40:42.372149 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:42.372157 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:42.372333 (+ 176us) server_negotiation.cc:407] Connection header received
0620 18:40:42.372523 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:42.372528 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:42.372613 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:42.372693 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:40:42.373228 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:42.374083 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:42.375964 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:42.376428 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:42.376547 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:42.377901 (+ 1354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:40:42.377910 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:40:42.377925 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:40:42.378532 (+ 607us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:40:42.378583 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:40:42.378605 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:40:42.378690 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:42.378868 (+ 178us) server_negotiation.cc:299] Negotiation successful
0620 18:40:42.378942 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260620 18:40:42.380075 10508 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:40:42.437005 10563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:42.436408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.1:36117 (local address 127.0.0.1:42528)
0620 18:40:42.436634 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:42.436784 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.1:36117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260620 18:40:42.438318 10563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:42.437788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.1:36117 (local address 127.0.0.1:42540)
0620 18:40:42.438037 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:42.438132 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.1:36117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
I20260620 18:40:42.502170 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 10386
I20260620 18:40:42.521345 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 10035
I20260620 18:40:42.522607 10096 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:40:42.630517 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 10035
I20260620 18:40:42.630544 10563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:42.629824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.1:36117 (local address 127.0.0.1:42554)
0620 18:40:42.630243 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:42.630376 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.1:36117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239}
2026-06-20T18:40:42Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2991 ms)
[----------] 6 tests from EncryptionPolicyTest (21760 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980527431848-22676-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:40:42 dist-test-slave-5lgp krb5kdc[10569](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:40:42 dist-test-slave-5lgp krb5kdc[10569](info): set up 2 sockets
Jun 20 18:40:42 dist-test-slave-5lgp krb5kdc[10569](info): commencing operation
krb5kdc: starting...
W20260620 18:40:43.741912 22676 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980527431848-22676-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980527431848-22676-0/krb5kdc/test-user.keytab.
Jun 20 18:40:43 dist-test-slave-5lgp krb5kdc[10569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:40:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:40:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980527431848-22676-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980527431848-22676-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 18:40:43.990231 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980527431848-22676-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980527431848-22676-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980527431848-22676-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44181
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980527431848-22676-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44181
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:40:44.314958 10582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:40:44.315250 10582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:40:44.315338 10582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:40:44.322419 10582 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:40:44.322520 10582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:40:44.322564 10582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:40:44.322625 10582 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:40:44.322669 10582 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:40:44.332161 10582 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980527431848-22676-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980527431848-22676-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44181
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980527431848-22676-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980527431848-22676-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:40:44.333441 10582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:40:44.334785 10582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:40:44.346975 10590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:40:44.347054 10588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:40:44.347023 10587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:40:44.347970 10582 server_base.cc:1034] running on GCE node
I20260620 18:40:44.348778 10582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:40:44.350409 10582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:40:44.351660 10582 hybrid_clock.cc:648] HybridClock initialized: now 1781980844351613 us; error 35 us; skew 500 ppm
I20260620 18:40:44.352035 10582 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:40:44 dist-test-slave-5lgp krb5kdc[10569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980844, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:40:44.355255 10582 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 18:40:44.356833 10582 webserver.cc:460] Webserver started at http://127.0.0.1:36681/ using document root <none> and password file <none>
I20260620 18:40:44.357606 10582 fs_manager.cc:359] Metadata directory not provided
I20260620 18:40:44.357756 10582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:40:44.358075 10582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:40:44.361119 10582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980527431848-22676-0/minicluster-data/master-0/data/instance:
uuid: "dcf6ced3017841f192c65cac76cb376a"
format_stamp: "Formatted at 2026-06-20 18:40:44 on dist-test-slave-5lgp"
server_key: "9105c05a6751c439c32e59e336dc9ef4"
server_key_iv: "2809c2cb9754d2171e71e32d58e1b3d6"
server_key_version: "encryptionkey@0"
I20260620 18:40:44.361966 10582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980527431848-22676-0/minicluster-data/master-0/wal/instance:
uuid: "dcf6ced3017841f192c65cac76cb376a"
format_stamp: "Formatted at 2026-06-20 18:40:44 on dist-test-slave-5lgp"
server_key: "9105c05a6751c439c32e59e336dc9ef4"
server_key_iv: "2809c2cb9754d2171e71e32d58e1b3d6"
server_key_version: "encryptionkey@0"
I20260620 18:40:44.367033 10582 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 18:40:44.370498 10597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:40:44.371907 10582 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 18:40:44.372107 10582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980527431848-22676-0/minicluster-data/master-0/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980527431848-22676-0/minicluster-data/master-0/wal
uuid: "dcf6ced3017841f192c65cac76cb376a"
format_stamp: "Formatted at 2026-06-20 18:40:44 on dist-test-slave-5lgp"
server_key: "9105c05a6751c439c32e59e336dc9ef4"
server_key_iv: "2809c2cb9754d2171e71e32d58e1b3d6"
server_key_version: "encryptionkey@0"
I20260620 18:40:44.372314 10582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980527431848-22676-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980527431848-22676-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980527431848-22676-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:40:44.396632 10582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:40:44.401010 10582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:40:44.401325 10582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:40:44.420367 10582 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44181
I20260620 18:40:44.420414 10648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44181 every 8 connection(s)
I20260620 18:40:44.421901 10582 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
I20260620 18:40:44.425118 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 10582
I20260620 18:40:44.425325 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980527431848-22676-0/minicluster-data/master-0/wal/instance
I20260620 18:40:44.426152 22676 external_mini_cluster.cc:1428] Setting key bb2fea704d7bee13e90473c91cf6b4de
I20260620 18:40:44.426316 10649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:40:44.440716 10649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcf6ced3017841f192c65cac76cb376a: Bootstrap starting.
Jun 20 18:40:44 dist-test-slave-5lgp krb5kdc[10569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 18:40:44.444603 10649 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dcf6ced3017841f192c65cac76cb376a: Neither blocks nor log segments found. Creating new log.
I20260620 18:40:44.446009 10649 log.cc:824] T 00000000000000000000000000000000 P dcf6ced3017841f192c65cac76cb376a: Log is configured to *not* fsync() on all Append() calls
I20260620 18:40:44.449539 10649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcf6ced3017841f192c65cac76cb376a: No bootstrap required, opened a new log
I20260620 18:40:44.451319 10652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:44.428364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54831 (local address 127.0.0.1:44181)
0620 18:40:44.429447 (+ 1083us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:44.429464 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:44.429833 (+ 369us) server_negotiation.cc:407] Connection header received
0620 18:40:44.431420 (+ 1587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:44.431452 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:44.431995 (+ 543us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:44.432462 (+ 467us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:40:44.433816 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:44.435720 (+ 1904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:44.436940 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:44.438069 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:44.438273 (+ 204us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:44.442457 (+ 4184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:40:44.442470 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:40:44.442495 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:40:44.442550 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:40:44.446397 (+ 3847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:40:44.447419 (+ 1022us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:40:44.447426 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:40:44.447435 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:40:44.447624 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:40:44.448064 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:40:44.448069 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:40:44.448071 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:40:44.448724 (+ 653us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:40:44.449075 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:44.449581 (+ 506us) server_negotiation.cc:299] Negotiation successful
0620 18:40:44.449916 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":752,"thread_start_us":477,"threads_started":1}
I20260620 18:40:44.455904 22676 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:40:44.458526 10649 raft_consensus.cc:348] T 00000000000000000000000000000000 P dcf6ced3017841f192c65cac76cb376a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcf6ced3017841f192c65cac76cb376a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44181 } }
I20260620 18:40:44.458904 10649 raft_consensus.cc:374] T 00000000000000000000000000000000 P dcf6ced3017841f192c65cac76cb376a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:40:44.459048 10649 raft_consensus.cc:729] T 00000000000000000000000000000000 P dcf6ced3017841f192c65cac76cb376a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcf6ced3017841f192c65cac76cb376a, State: Initialized, Role: FOLLOWER
I20260620 18:40:44.459653 10649 consensus_queue.cc:260] T 00000000000000000000000000000000 P dcf6ced3017841f192c65cac76cb376a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcf6ced3017841f192c65cac76cb376a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44181 } }
I20260620 18:40:44.459901 10649 raft_consensus.cc:388] T 00000000000000000000000000000000 P dcf6ced3017841f192c65cac76cb376a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:40:44.460049 10649 raft_consensus.cc:482] T 00000000000000000000000000000000 P dcf6ced3017841f192c65cac76cb376a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:40:44.460227 10649 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dcf6ced3017841f192c65cac76cb376a [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:40:44.462930 10649 raft_consensus.cc:504] T 00000000000000000000000000000000 P dcf6ced3017841f192c65cac76cb376a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcf6ced3017841f192c65cac76cb376a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44181 } }
I20260620 18:40:44.463478 10649 leader_election.cc:302] T 00000000000000000000000000000000 P dcf6ced3017841f192c65cac76cb376a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dcf6ced3017841f192c65cac76cb376a; no voters:
I20260620 18:40:44.464167 10649 leader_election.cc:288] T 00000000000000000000000000000000 P dcf6ced3017841f192c65cac76cb376a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:40:44.464275 10659 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dcf6ced3017841f192c65cac76cb376a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:40:44.465497 10659 raft_consensus.cc:686] T 00000000000000000000000000000000 P dcf6ced3017841f192c65cac76cb376a [term 1 LEADER]: Becoming Leader. State: Replica: dcf6ced3017841f192c65cac76cb376a, State: Running, Role: LEADER
I20260620 18:40:44.466111 10659 consensus_queue.cc:237] T 00000000000000000000000000000000 P dcf6ced3017841f192c65cac76cb376a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcf6ced3017841f192c65cac76cb376a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44181 } }
I20260620 18:40:44.466423 10649 sys_catalog.cc:564] T 00000000000000000000000000000000 P dcf6ced3017841f192c65cac76cb376a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:40:44.469333 10660 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcf6ced3017841f192c65cac76cb376a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dcf6ced3017841f192c65cac76cb376a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcf6ced3017841f192c65cac76cb376a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44181 } } }
I20260620 18:40:44.469702 10660 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcf6ced3017841f192c65cac76cb376a [sys.catalog]: This master's current role is: LEADER
I20260620 18:40:44.470189 10661 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcf6ced3017841f192c65cac76cb376a [sys.catalog]: SysCatalogTable state changed. Reason: New leader dcf6ced3017841f192c65cac76cb376a. Latest consensus state: current_term: 1 leader_uuid: "dcf6ced3017841f192c65cac76cb376a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcf6ced3017841f192c65cac76cb376a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44181 } } }
I20260620 18:40:44.470557 10661 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcf6ced3017841f192c65cac76cb376a [sys.catalog]: This master's current role is: LEADER
I20260620 18:40:44.478049 10665 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:40:44.480906 10652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:44.460937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52849 (local address 127.0.0.1:44181)
0620 18:40:44.461126 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:44.461132 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:44.461680 (+ 548us) server_negotiation.cc:407] Connection header received
0620 18:40:44.461824 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:44.461830 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:44.461910 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:44.462016 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:40:44.463458 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:44.464723 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:44.466522 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:44.467191 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:44.467329 (+ 138us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:44.471254 (+ 3925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:40:44.471267 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:40:44.471275 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:40:44.471329 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:40:44.475973 (+ 4644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:40:44.477357 (+ 1384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:40:44.477390 (+ 33us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:40:44.477394 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:40:44.477555 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:40:44.478134 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:40:44.478142 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:40:44.478146 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:40:44.478754 (+ 608us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:40:44.479956 (+ 1202us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:44.480196 (+ 240us) server_negotiation.cc:299] Negotiation successful
0620 18:40:44.480418 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 18:40:44.484803 10665 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:40:44.494376 10665 catalog_manager.cc:1347] Generated new cluster ID: 6aa12ca96fcc486e9d6900b3babf6d98
I20260620 18:40:44.494509 10665 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:40:44.519409 10665 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:40:44.520925 10665 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:40:44.533802 10665 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dcf6ced3017841f192c65cac76cb376a: Generated new TSK 0
I20260620 18:40:44.534780 10665 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:40:44.547436 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 10582
I20260620 18:40:44.548810 10644 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:40:44.643189 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 10582
W20260620 18:40:44.660694 10567 connection.cc:535] client connection to 127.0.0.1:44181 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44181): Connection reset by peer (error 104)
2026-06-20T18:40:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2003 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980527431848-22676-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:40:44 dist-test-slave-5lgp krb5kdc[10682](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:40:44 dist-test-slave-5lgp krb5kdc[10682](info): set up 2 sockets
Jun 20 18:40:44 dist-test-slave-5lgp krb5kdc[10682](info): commencing operation
krb5kdc: starting...
W20260620 18:40:45.769805 22676 mini_kdc.cc:121] Time spent starting KDC: real 1.090s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980527431848-22676-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980527431848-22676-0/krb5kdc/test-user.keytab.
Jun 20 18:40:45 dist-test-slave-5lgp krb5kdc[10682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:40:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:40:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980527431848-22676-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980527431848-22676-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 18:40:46.018343 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980527431848-22676-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980527431848-22676-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980527431848-22676-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43611
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980527431848-22676-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43611
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:40:46.345150 10695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:40:46.345511 10695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:40:46.345674 10695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:40:46.352777 10695 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:40:46.352926 10695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:40:46.353035 10695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:40:46.353148 10695 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:40:46.353236 10695 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:40:46.363287 10695 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980527431848-22676-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980527431848-22676-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43611
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980527431848-22676-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43611
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980527431848-22676-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:40:46.364671 10695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:40:46.366086 10695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:40:46.378273 10703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:40:46.378341 10700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:40:46.378381 10701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:40:46.379319 10695 server_base.cc:1034] running on GCE node
I20260620 18:40:46.380096 10695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:40:46.381615 10695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:40:46.382818 10695 hybrid_clock.cc:648] HybridClock initialized: now 1781980846382777 us; error 32 us; skew 500 ppm
I20260620 18:40:46.383142 10695 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:40:46 dist-test-slave-5lgp krb5kdc[10682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980846, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:40:46.392218 10695 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 18:40:46.393872 10695 webserver.cc:460] Webserver started at http://127.0.0.1:43423/ using document root <none> and password file <none>
I20260620 18:40:46.394537 10695 fs_manager.cc:359] Metadata directory not provided
I20260620 18:40:46.394639 10695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:40:46.394883 10695 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:40:46.397929 10695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980527431848-22676-0/minicluster-data/master-0/data/instance:
uuid: "04726e461e5a4db8bc378ea8123801e3"
format_stamp: "Formatted at 2026-06-20 18:40:46 on dist-test-slave-5lgp"
server_key: "543b955b2cd56cdb4e457d96d07115d9"
server_key_iv: "9f8cc7812dfbfd5159ea5e4cf0c55c18"
server_key_version: "encryptionkey@0"
I20260620 18:40:46.398639 10695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980527431848-22676-0/minicluster-data/master-0/wal/instance:
uuid: "04726e461e5a4db8bc378ea8123801e3"
format_stamp: "Formatted at 2026-06-20 18:40:46 on dist-test-slave-5lgp"
server_key: "543b955b2cd56cdb4e457d96d07115d9"
server_key_iv: "9f8cc7812dfbfd5159ea5e4cf0c55c18"
server_key_version: "encryptionkey@0"
I20260620 18:40:46.403841 10695 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:40:46.407371 10710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:40:46.408735 10695 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260620 18:40:46.408874 10695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980527431848-22676-0/minicluster-data/master-0/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980527431848-22676-0/minicluster-data/master-0/wal
uuid: "04726e461e5a4db8bc378ea8123801e3"
format_stamp: "Formatted at 2026-06-20 18:40:46 on dist-test-slave-5lgp"
server_key: "543b955b2cd56cdb4e457d96d07115d9"
server_key_iv: "9f8cc7812dfbfd5159ea5e4cf0c55c18"
server_key_version: "encryptionkey@0"
I20260620 18:40:46.409039 10695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980527431848-22676-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980527431848-22676-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980527431848-22676-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:40:46.433318 10695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:40:46.437433 10695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:40:46.437677 10695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:40:46.456177 10695 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43611
I20260620 18:40:46.456202 10761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43611 every 8 connection(s)
I20260620 18:40:46.457623 10695 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
I20260620 18:40:46.461218 10762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:40:46.462314 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 10695
I20260620 18:40:46.462517 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980527431848-22676-0/minicluster-data/master-0/wal/instance
I20260620 18:40:46.463076 22676 external_mini_cluster.cc:1428] Setting key 7e11bf7106ff46f1646f57bcfa5b3ff3
I20260620 18:40:46.474681 10762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04726e461e5a4db8bc378ea8123801e3: Bootstrap starting.
I20260620 18:40:46.476586 10765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:46.464849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38459 (local address 127.0.0.1:43611)
0620 18:40:46.465624 (+ 775us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:46.465639 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:46.465683 (+ 44us) server_negotiation.cc:407] Connection header received
0620 18:40:46.467389 (+ 1706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:46.467423 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:46.467805 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:46.468129 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:40:46.468733 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:46.470114 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:46.472058 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:46.472741 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:46.472885 (+ 144us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:46.474162 (+ 1277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:40:46.474169 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:40:46.474182 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:40:46.474721 (+ 539us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:40:46.474964 (+ 243us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:40:46.474997 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:40:46.475144 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:46.475425 (+ 281us) server_negotiation.cc:299] Negotiation successful
0620 18:40:46.475619 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":348,"threads_started":1}
I20260620 18:40:46.478052 10762 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04726e461e5a4db8bc378ea8123801e3: Neither blocks nor log segments found. Creating new log.
I20260620 18:40:46.479204 10762 log.cc:824] T 00000000000000000000000000000000 P 04726e461e5a4db8bc378ea8123801e3: Log is configured to *not* fsync() on all Append() calls
I20260620 18:40:46.480515 22676 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:40:46.482461 10762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04726e461e5a4db8bc378ea8123801e3: No bootstrap required, opened a new log
I20260620 18:40:46.492550 10765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:46.485761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60155 (local address 127.0.0.1:43611)
0620 18:40:46.486018 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:46.486027 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:46.486533 (+ 506us) server_negotiation.cc:407] Connection header received
0620 18:40:46.486690 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:46.486700 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:46.486779 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:46.486890 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:40:46.487376 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:46.488524 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:46.489890 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:46.490612 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:46.490754 (+ 142us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:46.491824 (+ 1070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:40:46.491830 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:40:46.491835 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:40:46.491855 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:40:46.491898 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:40:46.491920 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:40:46.492039 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:46.492180 (+ 141us) server_negotiation.cc:299] Negotiation successful
0620 18:40:46.492231 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 18:40:46.492738 10762 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04726e461e5a4db8bc378ea8123801e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04726e461e5a4db8bc378ea8123801e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43611 } }
I20260620 18:40:46.493127 10762 raft_consensus.cc:374] T 00000000000000000000000000000000 P 04726e461e5a4db8bc378ea8123801e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:40:46.493230 10762 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04726e461e5a4db8bc378ea8123801e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04726e461e5a4db8bc378ea8123801e3, State: Initialized, Role: FOLLOWER
I20260620 18:40:46.493850 10762 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04726e461e5a4db8bc378ea8123801e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04726e461e5a4db8bc378ea8123801e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43611 } }
I20260620 18:40:46.494117 10762 raft_consensus.cc:388] T 00000000000000000000000000000000 P 04726e461e5a4db8bc378ea8123801e3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:40:46.494277 10762 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04726e461e5a4db8bc378ea8123801e3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:40:46.494451 10762 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04726e461e5a4db8bc378ea8123801e3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:40:46.496564 10762 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04726e461e5a4db8bc378ea8123801e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04726e461e5a4db8bc378ea8123801e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43611 } }
I20260620 18:40:46.497058 10762 leader_election.cc:302] T 00000000000000000000000000000000 P 04726e461e5a4db8bc378ea8123801e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 04726e461e5a4db8bc378ea8123801e3; no voters:
I20260620 18:40:46.497700 10762 leader_election.cc:288] T 00000000000000000000000000000000 P 04726e461e5a4db8bc378ea8123801e3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:40:46.497843 10772 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04726e461e5a4db8bc378ea8123801e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:40:46.499145 10772 raft_consensus.cc:686] T 00000000000000000000000000000000 P 04726e461e5a4db8bc378ea8123801e3 [term 1 LEADER]: Becoming Leader. State: Replica: 04726e461e5a4db8bc378ea8123801e3, State: Running, Role: LEADER
I20260620 18:40:46.499635 10772 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04726e461e5a4db8bc378ea8123801e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04726e461e5a4db8bc378ea8123801e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43611 } }
I20260620 18:40:46.499850 10762 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04726e461e5a4db8bc378ea8123801e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:40:46.502735 10774 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04726e461e5a4db8bc378ea8123801e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04726e461e5a4db8bc378ea8123801e3. Latest consensus state: current_term: 1 leader_uuid: "04726e461e5a4db8bc378ea8123801e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04726e461e5a4db8bc378ea8123801e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43611 } } }
I20260620 18:40:46.503190 10774 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04726e461e5a4db8bc378ea8123801e3 [sys.catalog]: This master's current role is: LEADER
I20260620 18:40:46.503595 10773 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04726e461e5a4db8bc378ea8123801e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "04726e461e5a4db8bc378ea8123801e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04726e461e5a4db8bc378ea8123801e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43611 } } }
I20260620 18:40:46.503860 10773 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04726e461e5a4db8bc378ea8123801e3 [sys.catalog]: This master's current role is: LEADER
I20260620 18:40:46.505726 10781 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:40:46.510411 10781 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:40:46.518074 10781 catalog_manager.cc:1347] Generated new cluster ID: abb38f4f4bef499e8e6cbb420b92d85b
I20260620 18:40:46.518181 10781 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:40:46.549640 10781 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:40:46.551066 10781 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:40:46.558096 10781 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 04726e461e5a4db8bc378ea8123801e3: Generated new TSK 0
I20260620 18:40:46.558892 10781 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:40:46.564754 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 10695
W20260620 18:40:46.565853 10757 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:38459
W20260620 18:40:46.566614 22676 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 10695: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:40:46.566797 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 10695
W20260620 18:40:46.584259 10680 connection.cc:535] client connection to 127.0.0.1:43611 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43611): Connection reset by peer (error 104)
2026-06-20T18:40:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1923 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980527431848-22676-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:40:46 dist-test-slave-5lgp krb5kdc[10793](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:40:46 dist-test-slave-5lgp krb5kdc[10793](info): set up 2 sockets
Jun 20 18:40:46 dist-test-slave-5lgp krb5kdc[10793](info): commencing operation
krb5kdc: starting...
W20260620 18:40:47.673777 22676 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980527431848-22676-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980527431848-22676-0/krb5kdc/test-user.keytab.
Jun 20 18:40:47 dist-test-slave-5lgp krb5kdc[10793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980847, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:40:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:40:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980527431848-22676-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980527431848-22676-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 18:40:47.922657 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980527431848-22676-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980527431848-22676-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980527431848-22676-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33341
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980527431848-22676-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33341
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:40:48.251080 10806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:40:48.251416 10806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:40:48.251588 10806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:40:48.258806 10806 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:40:48.258962 10806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:40:48.259073 10806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:40:48.259187 10806 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:40:48.259271 10806 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:40:48.269217 10806 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980527431848-22676-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980527431848-22676-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33341
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980527431848-22676-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980527431848-22676-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:40:48.270532 10806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:40:48.271867 10806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:40:48.283830 10811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:40:48.283963 10814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:40:48.283896 10812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:40:48.284798 10806 server_base.cc:1034] running on GCE node
I20260620 18:40:48.285790 10806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:40:48.287156 10806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:40:48.288434 10806 hybrid_clock.cc:648] HybridClock initialized: now 1781980848288398 us; error 43 us; skew 500 ppm
I20260620 18:40:48.288781 10806 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:40:48 dist-test-slave-5lgp krb5kdc[10793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980848, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:40:48.291805 10806 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 18:40:48.293227 10806 webserver.cc:460] Webserver started at http://127.0.0.1:46469/ using document root <none> and password file <none>
I20260620 18:40:48.293947 10806 fs_manager.cc:359] Metadata directory not provided
I20260620 18:40:48.294095 10806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:40:48.294423 10806 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:40:48.297289 10806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980527431848-22676-0/minicluster-data/master-0/data/instance:
uuid: "c27eea9433934e6fa1208afd02a639fc"
format_stamp: "Formatted at 2026-06-20 18:40:48 on dist-test-slave-5lgp"
server_key: "130a94387efc9fab7c95dbe988bca2db"
server_key_iv: "b3a6384cab24d19f4196edace9433582"
server_key_version: "encryptionkey@0"
I20260620 18:40:48.298225 10806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980527431848-22676-0/minicluster-data/master-0/wal/instance:
uuid: "c27eea9433934e6fa1208afd02a639fc"
format_stamp: "Formatted at 2026-06-20 18:40:48 on dist-test-slave-5lgp"
server_key: "130a94387efc9fab7c95dbe988bca2db"
server_key_iv: "b3a6384cab24d19f4196edace9433582"
server_key_version: "encryptionkey@0"
I20260620 18:40:48.303258 10806 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:40:48.306525 10821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:40:48.307751 10806 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:40:48.307935 10806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980527431848-22676-0/minicluster-data/master-0/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980527431848-22676-0/minicluster-data/master-0/wal
uuid: "c27eea9433934e6fa1208afd02a639fc"
format_stamp: "Formatted at 2026-06-20 18:40:48 on dist-test-slave-5lgp"
server_key: "130a94387efc9fab7c95dbe988bca2db"
server_key_iv: "b3a6384cab24d19f4196edace9433582"
server_key_version: "encryptionkey@0"
I20260620 18:40:48.308121 10806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980527431848-22676-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980527431848-22676-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980527431848-22676-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:40:48.313702 10806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:40:48.317440 10806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:40:48.317710 10806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:40:48.335956 10806 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33341
I20260620 18:40:48.336035 10872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33341 every 8 connection(s)
I20260620 18:40:48.337268 10806 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
I20260620 18:40:48.341188 10873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:40:48.345242 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 10806
I20260620 18:40:48.345453 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980527431848-22676-0/minicluster-data/master-0/wal/instance
I20260620 18:40:48.345901 22676 external_mini_cluster.cc:1428] Setting key 3920be1254d6b58156bff1c3a29688f1
I20260620 18:40:48.351903 10873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c27eea9433934e6fa1208afd02a639fc: Bootstrap starting.
I20260620 18:40:48.353654 10876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:48.347506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38185 (local address 127.0.0.1:33341)
0620 18:40:48.348241 (+ 735us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:48.348252 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:48.348288 (+ 36us) server_negotiation.cc:407] Connection header received
0620 18:40:48.350440 (+ 2152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:48.350460 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:48.350761 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:48.351067 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:40:48.352155 (+ 1088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:40:48.352161 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:40:48.352172 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:40:48.352185 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:40:48.352243 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:40:48.352261 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:40:48.352441 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:48.352534 (+ 93us) server_negotiation.cc:299] Negotiation successful
0620 18:40:48.352710 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":344,"threads_started":1}
I20260620 18:40:48.355209 10873 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c27eea9433934e6fa1208afd02a639fc: Neither blocks nor log segments found. Creating new log.
I20260620 18:40:48.356441 10873 log.cc:824] T 00000000000000000000000000000000 P c27eea9433934e6fa1208afd02a639fc: Log is configured to *not* fsync() on all Append() calls
I20260620 18:40:48.357592 22676 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:40:48.359140 10873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c27eea9433934e6fa1208afd02a639fc: No bootstrap required, opened a new log
I20260620 18:40:48.364251 10876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:48.361797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58729 (local address 127.0.0.1:33341)
0620 18:40:48.361992 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:48.362000 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:48.362475 (+ 475us) server_negotiation.cc:407] Connection header received
0620 18:40:48.362643 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:48.362649 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:48.362688 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:48.362764 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:40:48.363610 (+ 846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:40:48.363614 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:40:48.363617 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:40:48.363629 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:40:48.363666 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:40:48.363679 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:40:48.363768 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:48.363897 (+ 129us) server_negotiation.cc:299] Negotiation successful
0620 18:40:48.363938 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 18:40:48.367642 10873 raft_consensus.cc:348] T 00000000000000000000000000000000 P c27eea9433934e6fa1208afd02a639fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c27eea9433934e6fa1208afd02a639fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33341 } }
I20260620 18:40:48.367985 10873 raft_consensus.cc:374] T 00000000000000000000000000000000 P c27eea9433934e6fa1208afd02a639fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:40:48.368094 10873 raft_consensus.cc:729] T 00000000000000000000000000000000 P c27eea9433934e6fa1208afd02a639fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c27eea9433934e6fa1208afd02a639fc, State: Initialized, Role: FOLLOWER
I20260620 18:40:48.368585 10873 consensus_queue.cc:260] T 00000000000000000000000000000000 P c27eea9433934e6fa1208afd02a639fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c27eea9433934e6fa1208afd02a639fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33341 } }
I20260620 18:40:48.368844 10873 raft_consensus.cc:388] T 00000000000000000000000000000000 P c27eea9433934e6fa1208afd02a639fc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:40:48.368979 10873 raft_consensus.cc:482] T 00000000000000000000000000000000 P c27eea9433934e6fa1208afd02a639fc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:40:48.369146 10873 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c27eea9433934e6fa1208afd02a639fc [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:40:48.371239 10873 raft_consensus.cc:504] T 00000000000000000000000000000000 P c27eea9433934e6fa1208afd02a639fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c27eea9433934e6fa1208afd02a639fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33341 } }
I20260620 18:40:48.371702 10873 leader_election.cc:302] T 00000000000000000000000000000000 P c27eea9433934e6fa1208afd02a639fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c27eea9433934e6fa1208afd02a639fc; no voters:
I20260620 18:40:48.372215 10873 leader_election.cc:288] T 00000000000000000000000000000000 P c27eea9433934e6fa1208afd02a639fc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:40:48.372313 10883 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c27eea9433934e6fa1208afd02a639fc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:40:48.373011 10883 raft_consensus.cc:686] T 00000000000000000000000000000000 P c27eea9433934e6fa1208afd02a639fc [term 1 LEADER]: Becoming Leader. State: Replica: c27eea9433934e6fa1208afd02a639fc, State: Running, Role: LEADER
I20260620 18:40:48.373576 10883 consensus_queue.cc:237] T 00000000000000000000000000000000 P c27eea9433934e6fa1208afd02a639fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c27eea9433934e6fa1208afd02a639fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33341 } }
I20260620 18:40:48.374272 10873 sys_catalog.cc:564] T 00000000000000000000000000000000 P c27eea9433934e6fa1208afd02a639fc [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:40:48.376336 10885 sys_catalog.cc:455] T 00000000000000000000000000000000 P c27eea9433934e6fa1208afd02a639fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader c27eea9433934e6fa1208afd02a639fc. Latest consensus state: current_term: 1 leader_uuid: "c27eea9433934e6fa1208afd02a639fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c27eea9433934e6fa1208afd02a639fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33341 } } }
I20260620 18:40:48.376832 10885 sys_catalog.cc:458] T 00000000000000000000000000000000 P c27eea9433934e6fa1208afd02a639fc [sys.catalog]: This master's current role is: LEADER
I20260620 18:40:48.377799 10891 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:40:48.377772 10884 sys_catalog.cc:455] T 00000000000000000000000000000000 P c27eea9433934e6fa1208afd02a639fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c27eea9433934e6fa1208afd02a639fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c27eea9433934e6fa1208afd02a639fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33341 } } }
I20260620 18:40:48.378643 10884 sys_catalog.cc:458] T 00000000000000000000000000000000 P c27eea9433934e6fa1208afd02a639fc [sys.catalog]: This master's current role is: LEADER
I20260620 18:40:48.384928 10891 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:40:48.393440 10891 catalog_manager.cc:1347] Generated new cluster ID: d95177f73e5d40c0864b6ab75d7f02bb
I20260620 18:40:48.393549 10891 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:40:48.416657 10891 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:40:48.417127 10891 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:40:48.428637 10891 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c27eea9433934e6fa1208afd02a639fc: Generated new TSK 0
I20260620 18:40:48.429587 10891 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:40:48.435509 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 10806
W20260620 18:40:48.436519 10868 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:38185
W20260620 18:40:48.437155 22676 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 10806: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:40:48.437302 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 10806
2026-06-20T18:40:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1869 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980527431848-22676-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:40:48 dist-test-slave-5lgp krb5kdc[10904](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:40:48 dist-test-slave-5lgp krb5kdc[10904](info): set up 2 sockets
Jun 20 18:40:48 dist-test-slave-5lgp krb5kdc[10904](info): commencing operation
krb5kdc: starting...
W20260620 18:40:49.547838 22676 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.007s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980527431848-22676-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980527431848-22676-0/krb5kdc/test-user.keytab.
Jun 20 18:40:49 dist-test-slave-5lgp krb5kdc[10904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980849, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:40:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:40:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980527431848-22676-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980527431848-22676-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 18:40:49.796989 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980527431848-22676-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980527431848-22676-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980527431848-22676-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34401
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980527431848-22676-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34401
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:40:50.124644 10917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:40:50.124913 10917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:40:50.125003 10917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:40:50.131956 10917 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:40:50.132052 10917 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:40:50.132097 10917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:40:50.132154 10917 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:40:50.132196 10917 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:40:50.141937 10917 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980527431848-22676-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980527431848-22676-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34401
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980527431848-22676-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34401
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980527431848-22676-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:40:50.143121 10917 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:40:50.144310 10917 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:40:50.155822 10922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:40:50.155889 10925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:40:50.155853 10923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:40:50.156739 10917 server_base.cc:1034] running on GCE node
I20260620 18:40:50.157555 10917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:40:50.158962 10917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:40:50.160188 10917 hybrid_clock.cc:648] HybridClock initialized: now 1781980850160157 us; error 42 us; skew 500 ppm
I20260620 18:40:50.160539 10917 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:40:50 dist-test-slave-5lgp krb5kdc[10904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980850, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:40:50.163595 10917 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 18:40:50.165125 10917 webserver.cc:460] Webserver started at http://127.0.0.1:36819/ using document root <none> and password file <none>
I20260620 18:40:50.165863 10917 fs_manager.cc:359] Metadata directory not provided
I20260620 18:40:50.166009 10917 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:40:50.166316 10917 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:40:50.169267 10917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980527431848-22676-0/minicluster-data/master-0/data/instance:
uuid: "2d816a6e8b0a4115a263e9db382781a3"
format_stamp: "Formatted at 2026-06-20 18:40:50 on dist-test-slave-5lgp"
server_key: "2e33634df0be8dd1e36ecd08e74507ef"
server_key_iv: "c65ad773ad19852cb9bab3f7b835f8dc"
server_key_version: "encryptionkey@0"
I20260620 18:40:50.170114 10917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980527431848-22676-0/minicluster-data/master-0/wal/instance:
uuid: "2d816a6e8b0a4115a263e9db382781a3"
format_stamp: "Formatted at 2026-06-20 18:40:50 on dist-test-slave-5lgp"
server_key: "2e33634df0be8dd1e36ecd08e74507ef"
server_key_iv: "c65ad773ad19852cb9bab3f7b835f8dc"
server_key_version: "encryptionkey@0"
I20260620 18:40:50.175419 10917 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:40:50.178910 10932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:40:50.180270 10917 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 18:40:50.180467 10917 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980527431848-22676-0/minicluster-data/master-0/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980527431848-22676-0/minicluster-data/master-0/wal
uuid: "2d816a6e8b0a4115a263e9db382781a3"
format_stamp: "Formatted at 2026-06-20 18:40:50 on dist-test-slave-5lgp"
server_key: "2e33634df0be8dd1e36ecd08e74507ef"
server_key_iv: "c65ad773ad19852cb9bab3f7b835f8dc"
server_key_version: "encryptionkey@0"
I20260620 18:40:50.180698 10917 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980527431848-22676-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980527431848-22676-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980527431848-22676-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:40:50.200134 10917 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:40:50.204210 10917 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:40:50.204519 10917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:40:50.223253 10917 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34401
I20260620 18:40:50.223289 10983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34401 every 8 connection(s)
I20260620 18:40:50.224704 10917 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
I20260620 18:40:50.228686 10984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:40:50.229671 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 10917
I20260620 18:40:50.229866 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980527431848-22676-0/minicluster-data/master-0/wal/instance
I20260620 18:40:50.230463 22676 external_mini_cluster.cc:1428] Setting key 04194967da94a7fbc944e722cd6f2dc5
I20260620 18:40:50.242175 10984 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d816a6e8b0a4115a263e9db382781a3: Bootstrap starting.
Jun 20 18:40:50 dist-test-slave-5lgp krb5kdc[10904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980849, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 18:40:50.245492 10984 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d816a6e8b0a4115a263e9db382781a3: Neither blocks nor log segments found. Creating new log.
I20260620 18:40:50.246562 10984 log.cc:824] T 00000000000000000000000000000000 P 2d816a6e8b0a4115a263e9db382781a3: Log is configured to *not* fsync() on all Append() calls
I20260620 18:40:50.249540 10984 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d816a6e8b0a4115a263e9db382781a3: No bootstrap required, opened a new log
I20260620 18:40:50.253633 10987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:50.232403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39047 (local address 127.0.0.1:34401)
0620 18:40:50.233214 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:50.233228 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:50.233272 (+ 44us) server_negotiation.cc:407] Connection header received
0620 18:40:50.235988 (+ 2716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:50.236015 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:50.236448 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:50.236860 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:40:50.238334 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:50.239614 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:50.240922 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:50.241652 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:50.241859 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:50.245924 (+ 4065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:40:50.245934 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:40:50.245948 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:40:50.245983 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:40:50.249237 (+ 3254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:40:50.250042 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:40:50.250050 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:40:50.250056 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:40:50.250232 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:40:50.250663 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:40:50.250667 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:40:50.250669 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:40:50.251212 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:40:50.251568 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:50.252077 (+ 509us) server_negotiation.cc:299] Negotiation successful
0620 18:40:50.252510 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":326,"threads_started":1}
I20260620 18:40:50.257557 22676 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:40:50.258253 10984 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d816a6e8b0a4115a263e9db382781a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d816a6e8b0a4115a263e9db382781a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34401 } }
I20260620 18:40:50.258612 10984 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2d816a6e8b0a4115a263e9db382781a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:40:50.258721 10984 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d816a6e8b0a4115a263e9db382781a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d816a6e8b0a4115a263e9db382781a3, State: Initialized, Role: FOLLOWER
I20260620 18:40:50.259320 10984 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d816a6e8b0a4115a263e9db382781a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d816a6e8b0a4115a263e9db382781a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34401 } }
I20260620 18:40:50.259569 10984 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2d816a6e8b0a4115a263e9db382781a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:40:50.259704 10984 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d816a6e8b0a4115a263e9db382781a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:40:50.259891 10984 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d816a6e8b0a4115a263e9db382781a3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:40:50.262465 10984 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d816a6e8b0a4115a263e9db382781a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d816a6e8b0a4115a263e9db382781a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34401 } }
I20260620 18:40:50.263007 10984 leader_election.cc:302] T 00000000000000000000000000000000 P 2d816a6e8b0a4115a263e9db382781a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d816a6e8b0a4115a263e9db382781a3; no voters:
I20260620 18:40:50.263664 10984 leader_election.cc:288] T 00000000000000000000000000000000 P 2d816a6e8b0a4115a263e9db382781a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:40:50.263808 10994 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d816a6e8b0a4115a263e9db382781a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:40:50.264479 10994 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2d816a6e8b0a4115a263e9db382781a3 [term 1 LEADER]: Becoming Leader. State: Replica: 2d816a6e8b0a4115a263e9db382781a3, State: Running, Role: LEADER
I20260620 18:40:50.265166 10994 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d816a6e8b0a4115a263e9db382781a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d816a6e8b0a4115a263e9db382781a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34401 } }
I20260620 18:40:50.267395 10984 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d816a6e8b0a4115a263e9db382781a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:40:50.268249 10996 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d816a6e8b0a4115a263e9db382781a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2d816a6e8b0a4115a263e9db382781a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d816a6e8b0a4115a263e9db382781a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34401 } } }
I20260620 18:40:50.268224 10995 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d816a6e8b0a4115a263e9db382781a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d816a6e8b0a4115a263e9db382781a3. Latest consensus state: current_term: 1 leader_uuid: "2d816a6e8b0a4115a263e9db382781a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d816a6e8b0a4115a263e9db382781a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34401 } } }
I20260620 18:40:50.268791 10996 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d816a6e8b0a4115a263e9db382781a3 [sys.catalog]: This master's current role is: LEADER
I20260620 18:40:50.268929 10995 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d816a6e8b0a4115a263e9db382781a3 [sys.catalog]: This master's current role is: LEADER
I20260620 18:40:50.269764 10998 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:40:50.280006 10998 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:40:50.287716 10987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:50.262358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50593 (local address 127.0.0.1:34401)
0620 18:40:50.262577 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:50.262583 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:50.263023 (+ 440us) server_negotiation.cc:407] Connection header received
0620 18:40:50.263183 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:50.263188 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:50.263267 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:50.263375 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:40:50.265337 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:50.266441 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:50.268428 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:50.269296 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:50.269467 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:50.273574 (+ 4107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:40:50.273596 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:40:50.273604 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:40:50.273656 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:40:50.283251 (+ 9595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:40:50.284286 (+ 1035us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:40:50.284295 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:40:50.284301 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:40:50.284509 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:40:50.285000 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:40:50.285008 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:40:50.285013 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:40:50.285486 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:40:50.285719 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:50.287105 (+ 1386us) server_negotiation.cc:299] Negotiation successful
0620 18:40:50.287332 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 18:40:50.296811 10998 catalog_manager.cc:1347] Generated new cluster ID: 2486f568bb9344bbb3b56e9420856de0
I20260620 18:40:50.296936 10998 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:40:50.312414 10998 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:40:50.313783 10998 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:40:50.322206 10998 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2d816a6e8b0a4115a263e9db382781a3: Generated new TSK 0
I20260620 18:40:50.323027 10998 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:40:50.329535 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 10917
I20260620 18:40:50.330981 10979 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:40:50.421833 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 10917
2026-06-20T18:40:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1985 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980527431848-22676-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:40:50 dist-test-slave-5lgp krb5kdc[11016](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:40:50 dist-test-slave-5lgp krb5kdc[11016](info): set up 2 sockets
Jun 20 18:40:50 dist-test-slave-5lgp krb5kdc[11016](info): commencing operation
krb5kdc: starting...
W20260620 18:40:51.529837 22676 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.008s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980527431848-22676-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980527431848-22676-0/krb5kdc/test-user.keytab.
Jun 20 18:40:51 dist-test-slave-5lgp krb5kdc[11016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980851, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:40:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:40:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980527431848-22676-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980527431848-22676-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 18:40:51.784646 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980527431848-22676-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980527431848-22676-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980527431848-22676-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34467
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980527431848-22676-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34467
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:40:52.115706 11029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:40:52.116062 11029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:40:52.116226 11029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:40:52.123347 11029 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:40:52.123504 11029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:40:52.123613 11029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:40:52.123724 11029 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:40:52.123799 11029 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:40:52.133361 11029 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980527431848-22676-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980527431848-22676-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34467
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980527431848-22676-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980527431848-22676-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:40:52.134703 11029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:40:52.135993 11029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:40:52.147552 11034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:40:52.147501 11037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:40:52.147637 11035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:40:52.148346 11029 server_base.cc:1034] running on GCE node
I20260620 18:40:52.149143 11029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:40:52.150559 11029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:40:52.151855 11029 hybrid_clock.cc:648] HybridClock initialized: now 1781980852151800 us; error 51 us; skew 500 ppm
I20260620 18:40:52.152249 11029 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:40:52 dist-test-slave-5lgp krb5kdc[11016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980852, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:40:52.161614 11029 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 18:40:52.163038 11029 webserver.cc:460] Webserver started at http://127.0.0.1:34291/ using document root <none> and password file <none>
I20260620 18:40:52.163666 11029 fs_manager.cc:359] Metadata directory not provided
I20260620 18:40:52.163776 11029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:40:52.164029 11029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:40:52.167070 11029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980527431848-22676-0/minicluster-data/master-0/data/instance:
uuid: "d278382f43ab44958780af8bf72841a7"
format_stamp: "Formatted at 2026-06-20 18:40:52 on dist-test-slave-5lgp"
server_key: "31eaadb2035ca9493ec27bbed58e9f9c"
server_key_iv: "20f2710aa2fe233f07dd14fd4218c3b1"
server_key_version: "encryptionkey@0"
I20260620 18:40:52.167768 11029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980527431848-22676-0/minicluster-data/master-0/wal/instance:
uuid: "d278382f43ab44958780af8bf72841a7"
format_stamp: "Formatted at 2026-06-20 18:40:52 on dist-test-slave-5lgp"
server_key: "31eaadb2035ca9493ec27bbed58e9f9c"
server_key_iv: "20f2710aa2fe233f07dd14fd4218c3b1"
server_key_version: "encryptionkey@0"
I20260620 18:40:52.172952 11029 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 18:40:52.176363 11044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:40:52.177884 11029 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 18:40:52.178093 11029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980527431848-22676-0/minicluster-data/master-0/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980527431848-22676-0/minicluster-data/master-0/wal
uuid: "d278382f43ab44958780af8bf72841a7"
format_stamp: "Formatted at 2026-06-20 18:40:52 on dist-test-slave-5lgp"
server_key: "31eaadb2035ca9493ec27bbed58e9f9c"
server_key_iv: "20f2710aa2fe233f07dd14fd4218c3b1"
server_key_version: "encryptionkey@0"
I20260620 18:40:52.178278 11029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980527431848-22676-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980527431848-22676-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980527431848-22676-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:40:52.202414 11029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:40:52.206251 11029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:40:52.206529 11029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:40:52.224244 11029 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34467
I20260620 18:40:52.224277 11095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34467 every 8 connection(s)
I20260620 18:40:52.225565 11029 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
I20260620 18:40:52.229064 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 11029
I20260620 18:40:52.229252 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980527431848-22676-0/minicluster-data/master-0/wal/instance
I20260620 18:40:52.229809 22676 external_mini_cluster.cc:1428] Setting key 1bc087982976836314e85194ffa4b5b6
I20260620 18:40:52.229858 11096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:40:52.242900 11099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:52.231586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46665 (local address 127.0.0.1:34467)
0620 18:40:52.232351 (+ 765us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:52.232373 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:52.232409 (+ 36us) server_negotiation.cc:407] Connection header received
0620 18:40:52.234366 (+ 1957us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:52.234386 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:52.234759 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:52.235081 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:40:52.235637 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:52.236892 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:52.238798 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:52.239422 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:52.239552 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:52.240752 (+ 1200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:40:52.240759 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:40:52.240770 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:40:52.241291 (+ 521us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:40:52.241417 (+ 126us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:40:52.241450 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:40:52.241537 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:52.241764 (+ 227us) server_negotiation.cc:299] Negotiation successful
0620 18:40:52.242047 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":378,"threads_started":1}
I20260620 18:40:52.243340 11096 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d278382f43ab44958780af8bf72841a7: Bootstrap starting.
I20260620 18:40:52.246418 22676 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:40:52.246646 11096 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d278382f43ab44958780af8bf72841a7: Neither blocks nor log segments found. Creating new log.
I20260620 18:40:52.247754 11096 log.cc:824] T 00000000000000000000000000000000 P d278382f43ab44958780af8bf72841a7: Log is configured to *not* fsync() on all Append() calls
I20260620 18:40:52.250910 11096 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d278382f43ab44958780af8bf72841a7: No bootstrap required, opened a new log
I20260620 18:40:52.258140 11099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:52.251427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36713 (local address 127.0.0.1:34467)
0620 18:40:52.251589 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:52.251593 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:52.252051 (+ 458us) server_negotiation.cc:407] Connection header received
0620 18:40:52.252246 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:52.252251 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:52.252338 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:52.252440 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:40:52.252934 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:52.253777 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:52.255680 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:52.256192 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:52.256287 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:52.257335 (+ 1048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:40:52.257339 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:40:52.257342 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:40:52.257360 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:40:52.257438 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:40:52.257462 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:40:52.257581 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:52.257718 (+ 137us) server_negotiation.cc:299] Negotiation successful
0620 18:40:52.257788 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 18:40:52.260380 11096 raft_consensus.cc:348] T 00000000000000000000000000000000 P d278382f43ab44958780af8bf72841a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d278382f43ab44958780af8bf72841a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34467 } }
I20260620 18:40:52.260794 11096 raft_consensus.cc:374] T 00000000000000000000000000000000 P d278382f43ab44958780af8bf72841a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:40:52.260906 11096 raft_consensus.cc:729] T 00000000000000000000000000000000 P d278382f43ab44958780af8bf72841a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d278382f43ab44958780af8bf72841a7, State: Initialized, Role: FOLLOWER
I20260620 18:40:52.261557 11096 consensus_queue.cc:260] T 00000000000000000000000000000000 P d278382f43ab44958780af8bf72841a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d278382f43ab44958780af8bf72841a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34467 } }
I20260620 18:40:52.261804 11096 raft_consensus.cc:388] T 00000000000000000000000000000000 P d278382f43ab44958780af8bf72841a7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:40:52.261962 11096 raft_consensus.cc:482] T 00000000000000000000000000000000 P d278382f43ab44958780af8bf72841a7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:40:52.262144 11096 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d278382f43ab44958780af8bf72841a7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:40:52.264201 11096 raft_consensus.cc:504] T 00000000000000000000000000000000 P d278382f43ab44958780af8bf72841a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d278382f43ab44958780af8bf72841a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34467 } }
I20260620 18:40:52.264707 11096 leader_election.cc:302] T 00000000000000000000000000000000 P d278382f43ab44958780af8bf72841a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d278382f43ab44958780af8bf72841a7; no voters:
I20260620 18:40:52.265269 11096 leader_election.cc:288] T 00000000000000000000000000000000 P d278382f43ab44958780af8bf72841a7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:40:52.265414 11106 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d278382f43ab44958780af8bf72841a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:40:52.266095 11106 raft_consensus.cc:686] T 00000000000000000000000000000000 P d278382f43ab44958780af8bf72841a7 [term 1 LEADER]: Becoming Leader. State: Replica: d278382f43ab44958780af8bf72841a7, State: Running, Role: LEADER
I20260620 18:40:52.266647 11106 consensus_queue.cc:237] T 00000000000000000000000000000000 P d278382f43ab44958780af8bf72841a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d278382f43ab44958780af8bf72841a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34467 } }
I20260620 18:40:52.267409 11096 sys_catalog.cc:564] T 00000000000000000000000000000000 P d278382f43ab44958780af8bf72841a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:40:52.269892 11107 sys_catalog.cc:455] T 00000000000000000000000000000000 P d278382f43ab44958780af8bf72841a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d278382f43ab44958780af8bf72841a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d278382f43ab44958780af8bf72841a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34467 } } }
I20260620 18:40:52.269831 11108 sys_catalog.cc:455] T 00000000000000000000000000000000 P d278382f43ab44958780af8bf72841a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d278382f43ab44958780af8bf72841a7. Latest consensus state: current_term: 1 leader_uuid: "d278382f43ab44958780af8bf72841a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d278382f43ab44958780af8bf72841a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34467 } } }
I20260620 18:40:52.270360 11107 sys_catalog.cc:458] T 00000000000000000000000000000000 P d278382f43ab44958780af8bf72841a7 [sys.catalog]: This master's current role is: LEADER
I20260620 18:40:52.270409 11108 sys_catalog.cc:458] T 00000000000000000000000000000000 P d278382f43ab44958780af8bf72841a7 [sys.catalog]: This master's current role is: LEADER
I20260620 18:40:52.271163 11113 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:40:52.276461 11113 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:40:52.285876 11113 catalog_manager.cc:1347] Generated new cluster ID: 740fb086cee64cc3bca94e3b57f2359c
I20260620 18:40:52.286006 11113 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:40:52.310633 11113 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:40:52.311998 11113 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:40:52.321336 11113 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d278382f43ab44958780af8bf72841a7: Generated new TSK 0
I20260620 18:40:52.322265 11113 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:40:52.338215 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 11029
W20260620 18:40:52.339720 11091 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:46665
W20260620 18:40:52.340437 22676 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 11029: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:40:52.340590 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 11029
2026-06-20T18:40:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1918 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980527431848-22676-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:40:52 dist-test-slave-5lgp krb5kdc[11127](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:40:52 dist-test-slave-5lgp krb5kdc[11127](info): set up 2 sockets
Jun 20 18:40:52 dist-test-slave-5lgp krb5kdc[11127](info): commencing operation
krb5kdc: starting...
W20260620 18:40:53.433655 22676 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.005s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980527431848-22676-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980527431848-22676-0/krb5kdc/test-user.keytab.
Jun 20 18:40:53 dist-test-slave-5lgp krb5kdc[11127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:40:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:40:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980527431848-22676-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980527431848-22676-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 18:40:53.680300 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980527431848-22676-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980527431848-22676-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980527431848-22676-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42765
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980527431848-22676-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42765
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:40:54.017874 11140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:40:54.018190 11140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:40:54.018388 11140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:40:54.025815 11140 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:40:54.025980 11140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:40:54.026069 11140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:40:54.026178 11140 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:40:54.026264 11140 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:40:54.036280 11140 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980527431848-22676-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980527431848-22676-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42765
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980527431848-22676-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42765
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980527431848-22676-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:40:54.037659 11140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:40:54.039017 11140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:40:54.051205 11145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:40:54.051185 11146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:40:54.051772 11148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:40:54.052276 11140 server_base.cc:1034] running on GCE node
I20260620 18:40:54.053143 11140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:40:54.054720 11140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:40:54.055940 11140 hybrid_clock.cc:648] HybridClock initialized: now 1781980854055892 us; error 38 us; skew 500 ppm
I20260620 18:40:54.056337 11140 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:40:54 dist-test-slave-5lgp krb5kdc[11127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980854, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:40:54.059791 11140 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 18:40:54.061311 11140 webserver.cc:460] Webserver started at http://127.0.0.1:36123/ using document root <none> and password file <none>
I20260620 18:40:54.062057 11140 fs_manager.cc:359] Metadata directory not provided
I20260620 18:40:54.062196 11140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:40:54.062520 11140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:40:54.065747 11140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980527431848-22676-0/minicluster-data/master-0/data/instance:
uuid: "52d8a451dd934579a91bc44eeec2979a"
format_stamp: "Formatted at 2026-06-20 18:40:54 on dist-test-slave-5lgp"
server_key: "affbbca205594655819ed2019fa66d36"
server_key_iv: "a267c9368bc9c5f0e030117f57d752cb"
server_key_version: "encryptionkey@0"
I20260620 18:40:54.066571 11140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980527431848-22676-0/minicluster-data/master-0/wal/instance:
uuid: "52d8a451dd934579a91bc44eeec2979a"
format_stamp: "Formatted at 2026-06-20 18:40:54 on dist-test-slave-5lgp"
server_key: "affbbca205594655819ed2019fa66d36"
server_key_iv: "a267c9368bc9c5f0e030117f57d752cb"
server_key_version: "encryptionkey@0"
I20260620 18:40:54.072283 11140 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 18:40:54.076046 11155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:40:54.077474 11140 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 18:40:54.077648 11140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980527431848-22676-0/minicluster-data/master-0/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980527431848-22676-0/minicluster-data/master-0/wal
uuid: "52d8a451dd934579a91bc44eeec2979a"
format_stamp: "Formatted at 2026-06-20 18:40:54 on dist-test-slave-5lgp"
server_key: "affbbca205594655819ed2019fa66d36"
server_key_iv: "a267c9368bc9c5f0e030117f57d752cb"
server_key_version: "encryptionkey@0"
I20260620 18:40:54.077831 11140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980527431848-22676-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980527431848-22676-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980527431848-22676-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:40:54.083570 11140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:40:54.087950 11140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:40:54.088266 11140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:40:54.106791 11140 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42765
I20260620 18:40:54.106873 11206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42765 every 8 connection(s)
I20260620 18:40:54.108227 11140 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
I20260620 18:40:54.112149 11207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:40:54.114394 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 11140
I20260620 18:40:54.114578 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980527431848-22676-0/minicluster-data/master-0/wal/instance
I20260620 18:40:54.115049 22676 external_mini_cluster.cc:1428] Setting key 85d196882f736c7fabb4f82bb58c471c
I20260620 18:40:54.124329 11210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:54.116957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38103 (local address 127.0.0.1:42765)
0620 18:40:54.117893 (+ 936us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:54.117922 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:54.117974 (+ 52us) server_negotiation.cc:407] Connection header received
0620 18:40:54.120389 (+ 2415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:54.120418 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:54.120849 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:54.121307 (+ 458us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:40:54.122328 (+ 1021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:40:54.122338 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:40:54.122360 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:40:54.122377 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:40:54.122447 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:40:54.122478 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:40:54.122710 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:54.122855 (+ 145us) server_negotiation.cc:299] Negotiation successful
0620 18:40:54.123069 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":688,"thread_start_us":447,"threads_started":1}
I20260620 18:40:54.125945 11207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52d8a451dd934579a91bc44eeec2979a: Bootstrap starting.
I20260620 18:40:54.127982 22676 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:40:54.129455 11207 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 52d8a451dd934579a91bc44eeec2979a: Neither blocks nor log segments found. Creating new log.
I20260620 18:40:54.130712 11207 log.cc:824] T 00000000000000000000000000000000 P 52d8a451dd934579a91bc44eeec2979a: Log is configured to *not* fsync() on all Append() calls
I20260620 18:40:54.133711 11207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52d8a451dd934579a91bc44eeec2979a: No bootstrap required, opened a new log
I20260620 18:40:54.136030 11210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:54.133412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52129 (local address 127.0.0.1:42765)
0620 18:40:54.133594 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:54.133600 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:54.133999 (+ 399us) server_negotiation.cc:407] Connection header received
0620 18:40:54.134116 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:54.134127 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:54.134198 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:54.134309 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:40:54.135300 (+ 991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:40:54.135309 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:40:54.135315 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:40:54.135334 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:40:54.135380 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:40:54.135405 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:40:54.135546 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:54.135662 (+ 116us) server_negotiation.cc:299] Negotiation successful
0620 18:40:54.135723 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 18:40:54.142606 11207 raft_consensus.cc:348] T 00000000000000000000000000000000 P 52d8a451dd934579a91bc44eeec2979a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52d8a451dd934579a91bc44eeec2979a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42765 } }
I20260620 18:40:54.142951 11207 raft_consensus.cc:374] T 00000000000000000000000000000000 P 52d8a451dd934579a91bc44eeec2979a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:40:54.143029 11207 raft_consensus.cc:729] T 00000000000000000000000000000000 P 52d8a451dd934579a91bc44eeec2979a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52d8a451dd934579a91bc44eeec2979a, State: Initialized, Role: FOLLOWER
I20260620 18:40:54.143528 11207 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52d8a451dd934579a91bc44eeec2979a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52d8a451dd934579a91bc44eeec2979a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42765 } }
I20260620 18:40:54.143728 11207 raft_consensus.cc:388] T 00000000000000000000000000000000 P 52d8a451dd934579a91bc44eeec2979a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:40:54.143824 11207 raft_consensus.cc:482] T 00000000000000000000000000000000 P 52d8a451dd934579a91bc44eeec2979a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:40:54.143944 11207 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 52d8a451dd934579a91bc44eeec2979a [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:40:54.146108 11207 raft_consensus.cc:504] T 00000000000000000000000000000000 P 52d8a451dd934579a91bc44eeec2979a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52d8a451dd934579a91bc44eeec2979a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42765 } }
I20260620 18:40:54.146538 11207 leader_election.cc:302] T 00000000000000000000000000000000 P 52d8a451dd934579a91bc44eeec2979a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 52d8a451dd934579a91bc44eeec2979a; no voters:
I20260620 18:40:54.147114 11207 leader_election.cc:288] T 00000000000000000000000000000000 P 52d8a451dd934579a91bc44eeec2979a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:40:54.147212 11217 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 52d8a451dd934579a91bc44eeec2979a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:40:54.147944 11217 raft_consensus.cc:686] T 00000000000000000000000000000000 P 52d8a451dd934579a91bc44eeec2979a [term 1 LEADER]: Becoming Leader. State: Replica: 52d8a451dd934579a91bc44eeec2979a, State: Running, Role: LEADER
I20260620 18:40:54.148518 11217 consensus_queue.cc:237] T 00000000000000000000000000000000 P 52d8a451dd934579a91bc44eeec2979a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52d8a451dd934579a91bc44eeec2979a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42765 } }
I20260620 18:40:54.149426 11207 sys_catalog.cc:564] T 00000000000000000000000000000000 P 52d8a451dd934579a91bc44eeec2979a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:40:54.151870 11218 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52d8a451dd934579a91bc44eeec2979a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "52d8a451dd934579a91bc44eeec2979a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52d8a451dd934579a91bc44eeec2979a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42765 } } }
I20260620 18:40:54.151849 11219 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52d8a451dd934579a91bc44eeec2979a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52d8a451dd934579a91bc44eeec2979a. Latest consensus state: current_term: 1 leader_uuid: "52d8a451dd934579a91bc44eeec2979a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52d8a451dd934579a91bc44eeec2979a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42765 } } }
I20260620 18:40:54.152323 11218 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52d8a451dd934579a91bc44eeec2979a [sys.catalog]: This master's current role is: LEADER
I20260620 18:40:54.152459 11219 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52d8a451dd934579a91bc44eeec2979a [sys.catalog]: This master's current role is: LEADER
I20260620 18:40:54.154572 11225 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:40:54.160116 11225 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:40:54.169505 11225 catalog_manager.cc:1347] Generated new cluster ID: cc3f1f29686a4ffa91ea126e0183faa5
I20260620 18:40:54.169641 11225 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:40:54.183311 11225 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:40:54.183734 11225 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:40:54.189515 11225 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 52d8a451dd934579a91bc44eeec2979a: Generated new TSK 0
I20260620 18:40:54.190269 11225 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:40:54.196617 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 11140
W20260620 18:40:54.197625 11202 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:38103
W20260620 18:40:54.198383 22676 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 11140: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:40:54.198547 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 11140
2026-06-20T18:40:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1857 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980527431848-22676-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:40:54 dist-test-slave-5lgp krb5kdc[11238](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:40:54 dist-test-slave-5lgp krb5kdc[11238](info): set up 2 sockets
Jun 20 18:40:54 dist-test-slave-5lgp krb5kdc[11238](info): commencing operation
krb5kdc: starting...
W20260620 18:40:55.309743 22676 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980527431848-22676-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980527431848-22676-0/krb5kdc/test-user.keytab.
Jun 20 18:40:55 dist-test-slave-5lgp krb5kdc[11238](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:40:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:40:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980527431848-22676-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980527431848-22676-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 18:40:55.556445 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980527431848-22676-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980527431848-22676-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980527431848-22676-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36193
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980527431848-22676-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36193
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:40:55.882959 11251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:40:55.883257 11251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:40:55.883420 11251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:40:55.890554 11251 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:40:55.890657 11251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:40:55.890766 11251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:40:55.890869 11251 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:40:55.890970 11251 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:40:55.900662 11251 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980527431848-22676-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980527431848-22676-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36193
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980527431848-22676-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980527431848-22676-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:40:55.902052 11251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:40:55.903416 11251 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:40:55.915690 11257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:40:55.915710 11256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:40:55.915781 11259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:40:55.916414 11251 server_base.cc:1034] running on GCE node
I20260620 18:40:55.917249 11251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:40:55.918797 11251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:40:55.920056 11251 hybrid_clock.cc:648] HybridClock initialized: now 1781980855920000 us; error 65 us; skew 500 ppm
I20260620 18:40:55.920537 11251 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:40:55 dist-test-slave-5lgp krb5kdc[11238](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980855, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:40:55.924639 11251 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 18:40:55.926180 11251 webserver.cc:460] Webserver started at http://127.0.0.1:33361/ using document root <none> and password file <none>
I20260620 18:40:55.926893 11251 fs_manager.cc:359] Metadata directory not provided
I20260620 18:40:55.927042 11251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:40:55.927361 11251 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:40:55.930507 11251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980527431848-22676-0/minicluster-data/master-0/data/instance:
uuid: "6925fa9f03ad4920881405c76f973882"
format_stamp: "Formatted at 2026-06-20 18:40:55 on dist-test-slave-5lgp"
server_key: "34c360d77e3b42599e91a0f4f8da344a"
server_key_iv: "be3acf085c727bfcb8136667633f4406"
server_key_version: "encryptionkey@0"
I20260620 18:40:55.931290 11251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980527431848-22676-0/minicluster-data/master-0/wal/instance:
uuid: "6925fa9f03ad4920881405c76f973882"
format_stamp: "Formatted at 2026-06-20 18:40:55 on dist-test-slave-5lgp"
server_key: "34c360d77e3b42599e91a0f4f8da344a"
server_key_iv: "be3acf085c727bfcb8136667633f4406"
server_key_version: "encryptionkey@0"
I20260620 18:40:55.936575 11251 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 18:40:55.940240 11266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:40:55.941646 11251 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:40:55.941869 11251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980527431848-22676-0/minicluster-data/master-0/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980527431848-22676-0/minicluster-data/master-0/wal
uuid: "6925fa9f03ad4920881405c76f973882"
format_stamp: "Formatted at 2026-06-20 18:40:55 on dist-test-slave-5lgp"
server_key: "34c360d77e3b42599e91a0f4f8da344a"
server_key_iv: "be3acf085c727bfcb8136667633f4406"
server_key_version: "encryptionkey@0"
I20260620 18:40:55.942083 11251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980527431848-22676-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980527431848-22676-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980527431848-22676-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:40:55.973201 11251 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:40:55.978008 11251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:40:55.978325 11251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:40:55.998152 11251 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36193
I20260620 18:40:55.998184 11317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36193 every 8 connection(s)
I20260620 18:40:55.999614 11251 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
I20260620 18:40:56.001545 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 11251
I20260620 18:40:56.001747 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980527431848-22676-0/minicluster-data/master-0/wal/instance
I20260620 18:40:56.002209 22676 external_mini_cluster.cc:1428] Setting key 1ee94afd54116873b4bb8aded2f01e60
I20260620 18:40:56.004187 11318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 18:40:56 dist-test-slave-5lgp krb5kdc[11238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 18:40:56.017784 11318 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6925fa9f03ad4920881405c76f973882: Bootstrap starting.
I20260620 18:40:56.021970 11318 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6925fa9f03ad4920881405c76f973882: Neither blocks nor log segments found. Creating new log.
I20260620 18:40:56.023331 11318 log.cc:824] T 00000000000000000000000000000000 P 6925fa9f03ad4920881405c76f973882: Log is configured to *not* fsync() on all Append() calls
I20260620 18:40:56.027032 11318 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6925fa9f03ad4920881405c76f973882: No bootstrap required, opened a new log
I20260620 18:40:56.028079 11321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:56.004251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.106:42251 (local address 127.0.0.1:36193)
0620 18:40:56.005101 (+ 850us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:56.005116 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:56.005168 (+ 52us) server_negotiation.cc:407] Connection header received
0620 18:40:56.007362 (+ 2194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:56.007388 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:56.007946 (+ 558us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:56.008402 (+ 456us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:40:56.009822 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:56.011875 (+ 2053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:56.013099 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:56.014134 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:56.014331 (+ 197us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:56.018745 (+ 4414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:40:56.018756 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:40:56.018774 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:40:56.018828 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:40:56.023032 (+ 4204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:40:56.024052 (+ 1020us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:40:56.024068 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:40:56.024074 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:40:56.024246 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:40:56.024892 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:40:56.024897 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:40:56.024900 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:40:56.025619 (+ 719us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:40:56.025947 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:56.026353 (+ 406us) server_negotiation.cc:299] Negotiation successful
0620 18:40:56.026695 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":324,"threads_started":1}
I20260620 18:40:56.032225 22676 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:40:56.035790 11318 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6925fa9f03ad4920881405c76f973882 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6925fa9f03ad4920881405c76f973882" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36193 } }
I20260620 18:40:56.036156 11318 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6925fa9f03ad4920881405c76f973882 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:40:56.036266 11318 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6925fa9f03ad4920881405c76f973882 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6925fa9f03ad4920881405c76f973882, State: Initialized, Role: FOLLOWER
I20260620 18:40:56.036989 11318 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6925fa9f03ad4920881405c76f973882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6925fa9f03ad4920881405c76f973882" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36193 } }
I20260620 18:40:56.037293 11318 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6925fa9f03ad4920881405c76f973882 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:40:56.037513 11318 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6925fa9f03ad4920881405c76f973882 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:40:56.037703 11318 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6925fa9f03ad4920881405c76f973882 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:40:56.039907 11318 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6925fa9f03ad4920881405c76f973882 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6925fa9f03ad4920881405c76f973882" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36193 } }
I20260620 18:40:56.040423 11318 leader_election.cc:302] T 00000000000000000000000000000000 P 6925fa9f03ad4920881405c76f973882 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6925fa9f03ad4920881405c76f973882; no voters:
I20260620 18:40:56.041118 11318 leader_election.cc:288] T 00000000000000000000000000000000 P 6925fa9f03ad4920881405c76f973882 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:40:56.041224 11328 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6925fa9f03ad4920881405c76f973882 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:40:56.042899 11328 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6925fa9f03ad4920881405c76f973882 [term 1 LEADER]: Becoming Leader. State: Replica: 6925fa9f03ad4920881405c76f973882, State: Running, Role: LEADER
I20260620 18:40:56.043550 11328 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6925fa9f03ad4920881405c76f973882 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6925fa9f03ad4920881405c76f973882" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36193 } }
I20260620 18:40:56.043705 11318 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6925fa9f03ad4920881405c76f973882 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:40:56.048347 11330 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6925fa9f03ad4920881405c76f973882 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6925fa9f03ad4920881405c76f973882. Latest consensus state: current_term: 1 leader_uuid: "6925fa9f03ad4920881405c76f973882" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6925fa9f03ad4920881405c76f973882" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36193 } } }
I20260620 18:40:56.048756 11330 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6925fa9f03ad4920881405c76f973882 [sys.catalog]: This master's current role is: LEADER
I20260620 18:40:56.049638 11329 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6925fa9f03ad4920881405c76f973882 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6925fa9f03ad4920881405c76f973882" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6925fa9f03ad4920881405c76f973882" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36193 } } }
I20260620 18:40:56.049916 11329 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6925fa9f03ad4920881405c76f973882 [sys.catalog]: This master's current role is: LEADER
I20260620 18:40:56.055910 11321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:56.036786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.106:60845 (local address 127.0.0.1:36193)
0620 18:40:56.036960 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:56.036965 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:56.037541 (+ 576us) server_negotiation.cc:407] Connection header received
0620 18:40:56.037722 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:56.037727 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:56.037803 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:56.037915 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:40:56.039273 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:56.040100 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:56.041691 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:56.042345 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:56.042494 (+ 149us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:56.045488 (+ 2994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:40:56.045495 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:40:56.045499 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:40:56.045544 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:40:56.051479 (+ 5935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:40:56.052741 (+ 1262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:40:56.052750 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:40:56.052756 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:40:56.052971 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:40:56.053655 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:40:56.053663 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:40:56.053669 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:40:56.054156 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:40:56.054554 (+ 398us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:56.055020 (+ 466us) server_negotiation.cc:299] Negotiation successful
0620 18:40:56.055275 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 18:40:56.061674 11336 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:40:56.066844 11336 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:40:56.075968 11336 catalog_manager.cc:1347] Generated new cluster ID: 7df59a47366d4a7bb565ddf713c79add
I20260620 18:40:56.076102 11336 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:40:56.102602 11336 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:40:56.104692 11336 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:40:56.116353 11336 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6925fa9f03ad4920881405c76f973882: Generated new TSK 0
I20260620 18:40:56.117483 11336 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:40:56.135205 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 11251
I20260620 18:40:56.137086 11313 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:40:56.230149 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 11251
W20260620 18:40:56.248389 11236 connection.cc:535] client connection to 127.0.0.1:36193 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36193): Connection reset by peer (error 104)
2026-06-20T18:40:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2033 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980527431848-22676-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:40:56 dist-test-slave-5lgp krb5kdc[11350](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:40:56 dist-test-slave-5lgp krb5kdc[11350](info): set up 2 sockets
Jun 20 18:40:56 dist-test-slave-5lgp krb5kdc[11350](info): commencing operation
krb5kdc: starting...
W20260620 18:40:57.338013 22676 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.010s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980527431848-22676-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980527431848-22676-0/krb5kdc/test-user.keytab.
Jun 20 18:40:57 dist-test-slave-5lgp krb5kdc[11350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980857, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:40:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:40:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980527431848-22676-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980527431848-22676-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 18:40:57.580637 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980527431848-22676-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980527431848-22676-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980527431848-22676-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34021
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980527431848-22676-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34021
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:40:57.908290 11363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:40:57.908581 11363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:40:57.908674 11363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:40:57.915881 11363 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:40:57.915990 11363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:40:57.916036 11363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:40:57.916088 11363 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:40:57.916132 11363 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:40:57.925957 11363 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980527431848-22676-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980527431848-22676-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34021
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980527431848-22676-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34021
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980527431848-22676-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:40:57.927241 11363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:40:57.928920 11363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:40:57.941605 11371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:40:57.941710 11368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:40:57.941686 11369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:40:57.942489 11363 server_base.cc:1034] running on GCE node
I20260620 18:40:57.943244 11363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:40:57.944612 11363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:40:57.945806 11363 hybrid_clock.cc:648] HybridClock initialized: now 1781980857945743 us; error 51 us; skew 500 ppm
I20260620 18:40:57.946113 11363 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:40:57 dist-test-slave-5lgp krb5kdc[11350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980857, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:40:57.954773 11363 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 18:40:57.956275 11363 webserver.cc:460] Webserver started at http://127.0.0.1:38575/ using document root <none> and password file <none>
I20260620 18:40:57.956970 11363 fs_manager.cc:359] Metadata directory not provided
I20260620 18:40:57.957114 11363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:40:57.957487 11363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:40:57.960434 11363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980527431848-22676-0/minicluster-data/master-0/data/instance:
uuid: "14f0fba2cbb448618ef300bfb8205474"
format_stamp: "Formatted at 2026-06-20 18:40:57 on dist-test-slave-5lgp"
server_key: "31539cc13cf77d9c944cf8cc61a68039"
server_key_iv: "5f7546aaf740eadb831e29445b625cd9"
server_key_version: "encryptionkey@0"
I20260620 18:40:57.961203 11363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980527431848-22676-0/minicluster-data/master-0/wal/instance:
uuid: "14f0fba2cbb448618ef300bfb8205474"
format_stamp: "Formatted at 2026-06-20 18:40:57 on dist-test-slave-5lgp"
server_key: "31539cc13cf77d9c944cf8cc61a68039"
server_key_iv: "5f7546aaf740eadb831e29445b625cd9"
server_key_version: "encryptionkey@0"
I20260620 18:40:57.966369 11363 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 18:40:57.969913 11378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:40:57.971204 11363 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 18:40:57.971377 11363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980527431848-22676-0/minicluster-data/master-0/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980527431848-22676-0/minicluster-data/master-0/wal
uuid: "14f0fba2cbb448618ef300bfb8205474"
format_stamp: "Formatted at 2026-06-20 18:40:57 on dist-test-slave-5lgp"
server_key: "31539cc13cf77d9c944cf8cc61a68039"
server_key_iv: "5f7546aaf740eadb831e29445b625cd9"
server_key_version: "encryptionkey@0"
I20260620 18:40:57.971578 11363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980527431848-22676-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980527431848-22676-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980527431848-22676-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:40:57.984318 11363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:40:57.988193 11363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:40:57.988477 11363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:40:58.006719 11363 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34021
I20260620 18:40:58.006742 11429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34021 every 8 connection(s)
I20260620 18:40:58.008205 11363 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
I20260620 18:40:58.011956 11430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:40:58.012895 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 11363
I20260620 18:40:58.013087 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980527431848-22676-0/minicluster-data/master-0/wal/instance
I20260620 18:40:58.013614 22676 external_mini_cluster.cc:1428] Setting key 1b79b6eb16dd57b6be66d2e64b8caa13
I20260620 18:40:58.025713 11430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14f0fba2cbb448618ef300bfb8205474: Bootstrap starting.
I20260620 18:40:58.027217 11433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:58.015401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.106:49649 (local address 127.0.0.1:34021)
0620 18:40:58.016178 (+ 777us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:58.016189 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:58.016230 (+ 41us) server_negotiation.cc:407] Connection header received
0620 18:40:58.017960 (+ 1730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:58.017983 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:58.018396 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:58.018721 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:40:58.019393 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:58.020779 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:58.022763 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:58.023463 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:58.023611 (+ 148us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:58.024920 (+ 1309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:40:58.024927 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:40:58.024940 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:40:58.025546 (+ 606us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:40:58.025628 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:40:58.025653 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:40:58.025771 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:58.026095 (+ 324us) server_negotiation.cc:299] Negotiation successful
0620 18:40:58.026285 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":359,"threads_started":1}
I20260620 18:40:58.029211 11430 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 14f0fba2cbb448618ef300bfb8205474: Neither blocks nor log segments found. Creating new log.
I20260620 18:40:58.030568 11430 log.cc:824] T 00000000000000000000000000000000 P 14f0fba2cbb448618ef300bfb8205474: Log is configured to *not* fsync() on all Append() calls
I20260620 18:40:58.031055 22676 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:40:58.033291 11430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14f0fba2cbb448618ef300bfb8205474: No bootstrap required, opened a new log
I20260620 18:40:58.042351 11433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:58.035565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.106:35013 (local address 127.0.0.1:34021)
0620 18:40:58.035792 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:58.035798 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:58.036055 (+ 257us) server_negotiation.cc:407] Connection header received
0620 18:40:58.036221 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:58.036225 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:58.036285 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:58.036364 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:40:58.036958 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:58.037827 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:58.039781 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:58.040222 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:58.040319 (+ 97us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:58.041546 (+ 1227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:40:58.041554 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:40:58.041557 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:40:58.041577 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:40:58.041630 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:40:58.041645 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:40:58.041735 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:58.041945 (+ 210us) server_negotiation.cc:299] Negotiation successful
0620 18:40:58.041991 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 18:40:58.043275 11430 raft_consensus.cc:348] T 00000000000000000000000000000000 P 14f0fba2cbb448618ef300bfb8205474 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14f0fba2cbb448618ef300bfb8205474" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34021 } }
I20260620 18:40:58.043689 11430 raft_consensus.cc:374] T 00000000000000000000000000000000 P 14f0fba2cbb448618ef300bfb8205474 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:40:58.043808 11430 raft_consensus.cc:729] T 00000000000000000000000000000000 P 14f0fba2cbb448618ef300bfb8205474 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14f0fba2cbb448618ef300bfb8205474, State: Initialized, Role: FOLLOWER
I20260620 18:40:58.044385 11430 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14f0fba2cbb448618ef300bfb8205474 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14f0fba2cbb448618ef300bfb8205474" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34021 } }
I20260620 18:40:58.044654 11430 raft_consensus.cc:388] T 00000000000000000000000000000000 P 14f0fba2cbb448618ef300bfb8205474 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:40:58.044804 11430 raft_consensus.cc:482] T 00000000000000000000000000000000 P 14f0fba2cbb448618ef300bfb8205474 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:40:58.044970 11430 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 14f0fba2cbb448618ef300bfb8205474 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:40:58.047164 11430 raft_consensus.cc:504] T 00000000000000000000000000000000 P 14f0fba2cbb448618ef300bfb8205474 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14f0fba2cbb448618ef300bfb8205474" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34021 } }
I20260620 18:40:58.047670 11430 leader_election.cc:302] T 00000000000000000000000000000000 P 14f0fba2cbb448618ef300bfb8205474 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 14f0fba2cbb448618ef300bfb8205474; no voters:
I20260620 18:40:58.048265 11430 leader_election.cc:288] T 00000000000000000000000000000000 P 14f0fba2cbb448618ef300bfb8205474 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:40:58.048415 11440 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 14f0fba2cbb448618ef300bfb8205474 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:40:58.049744 11440 raft_consensus.cc:686] T 00000000000000000000000000000000 P 14f0fba2cbb448618ef300bfb8205474 [term 1 LEADER]: Becoming Leader. State: Replica: 14f0fba2cbb448618ef300bfb8205474, State: Running, Role: LEADER
I20260620 18:40:58.050356 11440 consensus_queue.cc:237] T 00000000000000000000000000000000 P 14f0fba2cbb448618ef300bfb8205474 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14f0fba2cbb448618ef300bfb8205474" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34021 } }
I20260620 18:40:58.050746 11430 sys_catalog.cc:564] T 00000000000000000000000000000000 P 14f0fba2cbb448618ef300bfb8205474 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:40:58.053275 11442 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14f0fba2cbb448618ef300bfb8205474 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 14f0fba2cbb448618ef300bfb8205474. Latest consensus state: current_term: 1 leader_uuid: "14f0fba2cbb448618ef300bfb8205474" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14f0fba2cbb448618ef300bfb8205474" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34021 } } }
I20260620 18:40:58.053715 11442 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14f0fba2cbb448618ef300bfb8205474 [sys.catalog]: This master's current role is: LEADER
I20260620 18:40:58.053612 11441 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14f0fba2cbb448618ef300bfb8205474 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "14f0fba2cbb448618ef300bfb8205474" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14f0fba2cbb448618ef300bfb8205474" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34021 } } }
I20260620 18:40:58.053897 11441 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14f0fba2cbb448618ef300bfb8205474 [sys.catalog]: This master's current role is: LEADER
I20260620 18:40:58.056602 11448 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:40:58.062006 11448 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:40:58.070617 11448 catalog_manager.cc:1347] Generated new cluster ID: fbf23869ba4a418588f2db3d354ef310
I20260620 18:40:58.070727 11448 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:40:58.082268 11448 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:40:58.083704 11448 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:40:58.091054 11448 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 14f0fba2cbb448618ef300bfb8205474: Generated new TSK 0
I20260620 18:40:58.091861 11448 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:40:58.100369 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 11363
W20260620 18:40:58.101540 11425 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.106:49649
W20260620 18:40:58.102298 22676 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 11363: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:40:58.102458 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 11363
W20260620 18:40:58.119393 11348 connection.cc:535] client connection to 127.0.0.1:34021 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34021): Connection reset by peer (error 104)
2026-06-20T18:40:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1870 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980527431848-22676-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:40:58 dist-test-slave-5lgp krb5kdc[11461](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:40:58 dist-test-slave-5lgp krb5kdc[11461](info): set up 2 sockets
Jun 20 18:40:58 dist-test-slave-5lgp krb5kdc[11461](info): commencing operation
krb5kdc: starting...
W20260620 18:40:59.205739 22676 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.009s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980527431848-22676-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980527431848-22676-0/krb5kdc/test-user.keytab.
Jun 20 18:40:59 dist-test-slave-5lgp krb5kdc[11461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980859, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:40:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:40:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980527431848-22676-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980527431848-22676-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 18:40:59.464159 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980527431848-22676-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980527431848-22676-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980527431848-22676-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39927
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980527431848-22676-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39927
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:40:59.803411 11474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:40:59.803686 11474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:40:59.803880 11474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:40:59.810742 11474 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:40:59.810832 11474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:40:59.810949 11474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:40:59.811033 11474 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:40:59.811110 11474 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:40:59.820369 11474 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980527431848-22676-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980527431848-22676-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39927
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980527431848-22676-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39927
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980527431848-22676-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:40:59.821784 11474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:40:59.823024 11474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:40:59.834401 11479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:40:59.834357 11480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:40:59.834316 11482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:40:59.835206 11474 server_base.cc:1034] running on GCE node
I20260620 18:40:59.835983 11474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:40:59.837292 11474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:40:59.838550 11474 hybrid_clock.cc:648] HybridClock initialized: now 1781980859838517 us; error 66 us; skew 500 ppm
I20260620 18:40:59.838881 11474 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:40:59 dist-test-slave-5lgp krb5kdc[11461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980859, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:40:59.841997 11474 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 18:40:59.843344 11474 webserver.cc:460] Webserver started at http://127.0.0.1:36861/ using document root <none> and password file <none>
I20260620 18:40:59.843938 11474 fs_manager.cc:359] Metadata directory not provided
I20260620 18:40:59.844033 11474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:40:59.844266 11474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:40:59.847144 11474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980527431848-22676-0/minicluster-data/master-0/data/instance:
uuid: "1d4e1e173ff44585998e175321fe6ebf"
format_stamp: "Formatted at 2026-06-20 18:40:59 on dist-test-slave-5lgp"
server_key: "0988a54b6b1bd16016596802a8abe800"
server_key_iv: "1a2729024c73f3827a29befb2254a86b"
server_key_version: "encryptionkey@0"
I20260620 18:40:59.847831 11474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980527431848-22676-0/minicluster-data/master-0/wal/instance:
uuid: "1d4e1e173ff44585998e175321fe6ebf"
format_stamp: "Formatted at 2026-06-20 18:40:59 on dist-test-slave-5lgp"
server_key: "0988a54b6b1bd16016596802a8abe800"
server_key_iv: "1a2729024c73f3827a29befb2254a86b"
server_key_version: "encryptionkey@0"
I20260620 18:40:59.852654 11474 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 18:40:59.855970 11489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:40:59.857215 11474 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:40:59.857443 11474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980527431848-22676-0/minicluster-data/master-0/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980527431848-22676-0/minicluster-data/master-0/wal
uuid: "1d4e1e173ff44585998e175321fe6ebf"
format_stamp: "Formatted at 2026-06-20 18:40:59 on dist-test-slave-5lgp"
server_key: "0988a54b6b1bd16016596802a8abe800"
server_key_iv: "1a2729024c73f3827a29befb2254a86b"
server_key_version: "encryptionkey@0"
I20260620 18:40:59.857656 11474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980527431848-22676-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980527431848-22676-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980527431848-22676-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:40:59.863065 11474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:40:59.866940 11474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:40:59.867226 11474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:40:59.885150 11474 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39927
I20260620 18:40:59.885190 11540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39927 every 8 connection(s)
I20260620 18:40:59.886586 11474 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
I20260620 18:40:59.887579 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 11474
I20260620 18:40:59.887758 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980527431848-22676-0/minicluster-data/master-0/wal/instance
I20260620 18:40:59.888314 22676 external_mini_cluster.cc:1428] Setting key 23a28f614131fb4a3c7342288281c22a
I20260620 18:40:59.891156 11541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:40:59.895839 11544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:59.890084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.106:49719 (local address 127.0.0.1:39927)
0620 18:40:59.890836 (+ 752us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:59.890847 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:59.890885 (+ 38us) server_negotiation.cc:407] Connection header received
0620 18:40:59.892516 (+ 1631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:59.892538 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:59.892849 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:59.893214 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:40:59.894298 (+ 1084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:40:59.894304 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:40:59.894316 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:40:59.894328 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:40:59.894388 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:40:59.894406 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:40:59.894647 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:59.894792 (+ 145us) server_negotiation.cc:299] Negotiation successful
0620 18:40:59.894962 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":331,"threads_started":1}
I20260620 18:40:59.899394 22676 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:40:59.902392 11541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d4e1e173ff44585998e175321fe6ebf: Bootstrap starting.
I20260620 18:40:59.906252 11541 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d4e1e173ff44585998e175321fe6ebf: Neither blocks nor log segments found. Creating new log.
I20260620 18:40:59.906337 11544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:59.903966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.106:37247 (local address 127.0.0.1:39927)
0620 18:40:59.904166 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:59.904172 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:59.904415 (+ 243us) server_negotiation.cc:407] Connection header received
0620 18:40:59.904584 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:59.904587 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:59.904635 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:59.904712 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:40:59.905670 (+ 958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:40:59.905674 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:40:59.905677 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:40:59.905691 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:40:59.905729 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:40:59.905744 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:40:59.905832 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:59.905982 (+ 150us) server_negotiation.cc:299] Negotiation successful
0620 18:40:59.906036 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260620 18:40:59.907404 11541 log.cc:824] T 00000000000000000000000000000000 P 1d4e1e173ff44585998e175321fe6ebf: Log is configured to *not* fsync() on all Append() calls
I20260620 18:40:59.910034 11541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d4e1e173ff44585998e175321fe6ebf: No bootstrap required, opened a new log
I20260620 18:40:59.917019 11541 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d4e1e173ff44585998e175321fe6ebf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d4e1e173ff44585998e175321fe6ebf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39927 } }
I20260620 18:40:59.917479 11541 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d4e1e173ff44585998e175321fe6ebf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:40:59.917619 11541 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d4e1e173ff44585998e175321fe6ebf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d4e1e173ff44585998e175321fe6ebf, State: Initialized, Role: FOLLOWER
I20260620 18:40:59.918254 11541 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d4e1e173ff44585998e175321fe6ebf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d4e1e173ff44585998e175321fe6ebf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39927 } }
I20260620 18:40:59.918486 11541 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1d4e1e173ff44585998e175321fe6ebf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:40:59.918637 11541 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d4e1e173ff44585998e175321fe6ebf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:40:59.918799 11541 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d4e1e173ff44585998e175321fe6ebf [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:40:59.920953 11541 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d4e1e173ff44585998e175321fe6ebf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d4e1e173ff44585998e175321fe6ebf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39927 } }
I20260620 18:40:59.921586 11541 leader_election.cc:302] T 00000000000000000000000000000000 P 1d4e1e173ff44585998e175321fe6ebf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d4e1e173ff44585998e175321fe6ebf; no voters:
I20260620 18:40:59.922207 11541 leader_election.cc:288] T 00000000000000000000000000000000 P 1d4e1e173ff44585998e175321fe6ebf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:40:59.922355 11551 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d4e1e173ff44585998e175321fe6ebf [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:40:59.923027 11551 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d4e1e173ff44585998e175321fe6ebf [term 1 LEADER]: Becoming Leader. State: Replica: 1d4e1e173ff44585998e175321fe6ebf, State: Running, Role: LEADER
I20260620 18:40:59.923595 11551 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d4e1e173ff44585998e175321fe6ebf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d4e1e173ff44585998e175321fe6ebf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39927 } }
I20260620 18:40:59.924309 11541 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d4e1e173ff44585998e175321fe6ebf [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:40:59.926878 11553 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d4e1e173ff44585998e175321fe6ebf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d4e1e173ff44585998e175321fe6ebf. Latest consensus state: current_term: 1 leader_uuid: "1d4e1e173ff44585998e175321fe6ebf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d4e1e173ff44585998e175321fe6ebf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39927 } } }
I20260620 18:40:59.926853 11552 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d4e1e173ff44585998e175321fe6ebf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1d4e1e173ff44585998e175321fe6ebf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d4e1e173ff44585998e175321fe6ebf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39927 } } }
I20260620 18:40:59.927392 11553 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d4e1e173ff44585998e175321fe6ebf [sys.catalog]: This master's current role is: LEADER
I20260620 18:40:59.927434 11552 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d4e1e173ff44585998e175321fe6ebf [sys.catalog]: This master's current role is: LEADER
I20260620 18:40:59.930971 11557 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:40:59.936434 11557 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:40:59.945580 11557 catalog_manager.cc:1347] Generated new cluster ID: 14063ef2173d4d3da7717853d8af15d0
I20260620 18:40:59.945698 11557 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:40:59.957048 11557 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:40:59.957526 11557 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:40:59.964146 11557 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1d4e1e173ff44585998e175321fe6ebf: Generated new TSK 0
I20260620 18:40:59.965070 11557 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:40:59.973688 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 11474
W20260620 18:40:59.974788 11536 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.106:49719
W20260620 18:40:59.975659 22676 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 11474: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:40:59.975791 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 11474
2026-06-20T18:40:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1873 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980527431848-22676-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:41:00 dist-test-slave-5lgp krb5kdc[11572](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:41:00 dist-test-slave-5lgp krb5kdc[11572](info): set up 2 sockets
Jun 20 18:41:00 dist-test-slave-5lgp krb5kdc[11572](info): commencing operation
krb5kdc: starting...
W20260620 18:41:01.089812 22676 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980527431848-22676-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980527431848-22676-0/krb5kdc/test-user.keytab.
Jun 20 18:41:01 dist-test-slave-5lgp krb5kdc[11572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980861, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:41:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:41:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980527431848-22676-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980527431848-22676-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 18:41:01.354712 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980527431848-22676-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980527431848-22676-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980527431848-22676-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44749
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980527431848-22676-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44749
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:41:01.686573 11585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:41:01.686937 11585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:41:01.687100 11585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:41:01.694403 11585 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:41:01.694552 11585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:41:01.694655 11585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:41:01.694759 11585 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:41:01.694841 11585 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:41:01.704290 11585 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980527431848-22676-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980527431848-22676-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44749
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980527431848-22676-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44749
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980527431848-22676-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:41:01.705593 11585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:41:01.706904 11585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:41:01.718858 11591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:01.718791 11593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:01.719091 11590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:41:01.720194 11585 server_base.cc:1034] running on GCE node
I20260620 18:41:01.720973 11585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:41:01.722321 11585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:41:01.723518 11585 hybrid_clock.cc:648] HybridClock initialized: now 1781980861723456 us; error 53 us; skew 500 ppm
I20260620 18:41:01.723841 11585 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:41:01 dist-test-slave-5lgp krb5kdc[11572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980861, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:41:01.726979 11585 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 18:41:01.728410 11585 webserver.cc:460] Webserver started at http://127.0.0.1:33063/ using document root <none> and password file <none>
I20260620 18:41:01.729071 11585 fs_manager.cc:359] Metadata directory not provided
I20260620 18:41:01.729226 11585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:41:01.729578 11585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:41:01.732484 11585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980527431848-22676-0/minicluster-data/master-0/data/instance:
uuid: "3a4d80c5b0ec4443b7296b7b4f36bf74"
format_stamp: "Formatted at 2026-06-20 18:41:01 on dist-test-slave-5lgp"
server_key: "6fc15230a13adcecdd126b23e551177a"
server_key_iv: "9aed2020acf5e19ebfbb8b497e5b8fcb"
server_key_version: "encryptionkey@0"
I20260620 18:41:01.733278 11585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980527431848-22676-0/minicluster-data/master-0/wal/instance:
uuid: "3a4d80c5b0ec4443b7296b7b4f36bf74"
format_stamp: "Formatted at 2026-06-20 18:41:01 on dist-test-slave-5lgp"
server_key: "6fc15230a13adcecdd126b23e551177a"
server_key_iv: "9aed2020acf5e19ebfbb8b497e5b8fcb"
server_key_version: "encryptionkey@0"
I20260620 18:41:01.739068 11585 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260620 18:41:01.742524 11600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:01.743840 11585 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 18:41:01.744040 11585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980527431848-22676-0/minicluster-data/master-0/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980527431848-22676-0/minicluster-data/master-0/wal
uuid: "3a4d80c5b0ec4443b7296b7b4f36bf74"
format_stamp: "Formatted at 2026-06-20 18:41:01 on dist-test-slave-5lgp"
server_key: "6fc15230a13adcecdd126b23e551177a"
server_key_iv: "9aed2020acf5e19ebfbb8b497e5b8fcb"
server_key_version: "encryptionkey@0"
I20260620 18:41:01.744243 11585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980527431848-22676-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980527431848-22676-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980527431848-22676-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:41:01.762391 11585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:41:01.766533 11585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:41:01.766834 11585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:41:01.785622 11585 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44749
I20260620 18:41:01.785652 11651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44749 every 8 connection(s)
I20260620 18:41:01.787031 11585 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
I20260620 18:41:01.789170 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 11585
I20260620 18:41:01.789345 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980527431848-22676-0/minicluster-data/master-0/wal/instance
I20260620 18:41:01.789819 22676 external_mini_cluster.cc:1428] Setting key 45eb781a8b10f6c6f7384109cf7b3d50
I20260620 18:41:01.791265 11652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 18:41:01 dist-test-slave-5lgp krb5kdc[11572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980861, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 18:41:01.804677 11652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a4d80c5b0ec4443b7296b7b4f36bf74: Bootstrap starting.
I20260620 18:41:01.808660 11652 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a4d80c5b0ec4443b7296b7b4f36bf74: Neither blocks nor log segments found. Creating new log.
I20260620 18:41:01.810058 11652 log.cc:824] T 00000000000000000000000000000000 P 3a4d80c5b0ec4443b7296b7b4f36bf74: Log is configured to *not* fsync() on all Append() calls
I20260620 18:41:01.813053 11652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a4d80c5b0ec4443b7296b7b4f36bf74: No bootstrap required, opened a new log
I20260620 18:41:01.814455 11655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:01.791638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.106:42095 (local address 127.0.0.1:44749)
0620 18:41:01.792392 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:01.792407 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:01.792457 (+ 50us) server_negotiation.cc:407] Connection header received
0620 18:41:01.794532 (+ 2075us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:01.794559 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:01.795098 (+ 539us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:01.795581 (+ 483us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:01.796920 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:01.798957 (+ 2037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:01.800213 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:01.801264 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:01.801475 (+ 211us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:01.805872 (+ 4397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:01.805883 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:01.805902 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:01.805957 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:01.810032 (+ 4075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:01.810920 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:01.810928 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:01.810937 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:01.811169 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:01.811707 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:01.811711 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:01.811713 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:01.812205 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:01.812551 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:01.813051 (+ 500us) server_negotiation.cc:299] Negotiation successful
0620 18:41:01.813431 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":303,"threads_started":1}
I20260620 18:41:01.818576 22676 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:41:01.821671 11652 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a4d80c5b0ec4443b7296b7b4f36bf74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a4d80c5b0ec4443b7296b7b4f36bf74" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44749 } }
I20260620 18:41:01.822042 11652 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3a4d80c5b0ec4443b7296b7b4f36bf74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:41:01.822149 11652 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a4d80c5b0ec4443b7296b7b4f36bf74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a4d80c5b0ec4443b7296b7b4f36bf74, State: Initialized, Role: FOLLOWER
I20260620 18:41:01.822757 11652 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a4d80c5b0ec4443b7296b7b4f36bf74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a4d80c5b0ec4443b7296b7b4f36bf74" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44749 } }
I20260620 18:41:01.823071 11652 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3a4d80c5b0ec4443b7296b7b4f36bf74 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:41:01.823246 11652 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a4d80c5b0ec4443b7296b7b4f36bf74 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:41:01.823522 11652 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a4d80c5b0ec4443b7296b7b4f36bf74 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:41:01.826198 11652 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a4d80c5b0ec4443b7296b7b4f36bf74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a4d80c5b0ec4443b7296b7b4f36bf74" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44749 } }
I20260620 18:41:01.826772 11652 leader_election.cc:302] T 00000000000000000000000000000000 P 3a4d80c5b0ec4443b7296b7b4f36bf74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3a4d80c5b0ec4443b7296b7b4f36bf74; no voters:
I20260620 18:41:01.827358 11652 leader_election.cc:288] T 00000000000000000000000000000000 P 3a4d80c5b0ec4443b7296b7b4f36bf74 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:41:01.827491 11662 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3a4d80c5b0ec4443b7296b7b4f36bf74 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:41:01.828141 11662 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3a4d80c5b0ec4443b7296b7b4f36bf74 [term 1 LEADER]: Becoming Leader. State: Replica: 3a4d80c5b0ec4443b7296b7b4f36bf74, State: Running, Role: LEADER
I20260620 18:41:01.828850 11662 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a4d80c5b0ec4443b7296b7b4f36bf74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a4d80c5b0ec4443b7296b7b4f36bf74" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44749 } }
I20260620 18:41:01.831101 11652 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a4d80c5b0ec4443b7296b7b4f36bf74 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:41:01.837554 11664 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a4d80c5b0ec4443b7296b7b4f36bf74 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3a4d80c5b0ec4443b7296b7b4f36bf74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a4d80c5b0ec4443b7296b7b4f36bf74" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44749 } } }
I20260620 18:41:01.838099 11664 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a4d80c5b0ec4443b7296b7b4f36bf74 [sys.catalog]: This master's current role is: LEADER
I20260620 18:41:01.837363 11663 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a4d80c5b0ec4443b7296b7b4f36bf74 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a4d80c5b0ec4443b7296b7b4f36bf74. Latest consensus state: current_term: 1 leader_uuid: "3a4d80c5b0ec4443b7296b7b4f36bf74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a4d80c5b0ec4443b7296b7b4f36bf74" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44749 } } }
I20260620 18:41:01.839082 11663 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a4d80c5b0ec4443b7296b7b4f36bf74 [sys.catalog]: This master's current role is: LEADER
I20260620 18:41:01.843909 11655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:01.823126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.106:53355 (local address 127.0.0.1:44749)
0620 18:41:01.823348 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:01.823356 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:01.823679 (+ 323us) server_negotiation.cc:407] Connection header received
0620 18:41:01.823856 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:01.823861 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:01.823947 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:01.824049 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:01.825385 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:01.826548 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:01.828470 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:01.829148 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:01.829248 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:01.833912 (+ 4664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:01.833926 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:01.833934 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:01.834002 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:01.838286 (+ 4284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:01.839658 (+ 1372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:01.839668 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:01.839674 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:01.839957 (+ 283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:01.840718 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:01.840727 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:01.840733 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:01.841227 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:01.841631 (+ 404us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:01.843152 (+ 1521us) server_negotiation.cc:299] Negotiation successful
0620 18:41:01.843408 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 18:41:01.851707 11673 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:41:01.856366 11673 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:41:01.865161 11673 catalog_manager.cc:1347] Generated new cluster ID: 1ff0d9730c1342179a008d9af4d1acaa
I20260620 18:41:01.865272 11673 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:41:01.899586 11673 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:41:01.900924 11673 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:41:01.915097 11673 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3a4d80c5b0ec4443b7296b7b4f36bf74: Generated new TSK 0
I20260620 18:41:01.916145 11673 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:41:01.933573 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 11585
I20260620 18:41:01.934826 11647 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:41:02.030809 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 11585
W20260620 18:41:02.048537 11570 connection.cc:535] client connection to 127.0.0.1:44749 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44749): Connection reset by peer (error 104)
2026-06-20T18:41:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2055 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980527431848-22676-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:41:02 dist-test-slave-5lgp krb5kdc[11684](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:41:02 dist-test-slave-5lgp krb5kdc[11684](info): set up 2 sockets
Jun 20 18:41:02 dist-test-slave-5lgp krb5kdc[11684](info): commencing operation
krb5kdc: starting...
W20260620 18:41:03.129967 22676 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.006s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980527431848-22676-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980527431848-22676-0/krb5kdc/test-user.keytab.
Jun 20 18:41:03 dist-test-slave-5lgp krb5kdc[11684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980863, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:41:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:41:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980527431848-22676-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980527431848-22676-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 18:41:03.398767 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980527431848-22676-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980527431848-22676-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980527431848-22676-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40019
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980527431848-22676-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40019
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:41:03.736163 11697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:41:03.736461 11697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:41:03.736635 11697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:41:03.744042 11697 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:41:03.744189 11697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:41:03.744303 11697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:41:03.744418 11697 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:41:03.744511 11697 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:41:03.755234 11697 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980527431848-22676-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980527431848-22676-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40019
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980527431848-22676-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980527431848-22676-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:41:03.756552 11697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:41:03.757925 11697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:41:03.770375 11703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:03.770453 11702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:03.770431 11705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:41:03.771591 11697 server_base.cc:1034] running on GCE node
I20260620 18:41:03.772339 11697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:41:03.773763 11697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:41:03.774955 11697 hybrid_clock.cc:648] HybridClock initialized: now 1781980863774899 us; error 49 us; skew 500 ppm
I20260620 18:41:03.775292 11697 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:41:03 dist-test-slave-5lgp krb5kdc[11684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980863, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:41:03.778455 11697 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 18:41:03.779925 11697 webserver.cc:460] Webserver started at http://127.0.0.1:40521/ using document root <none> and password file <none>
I20260620 18:41:03.780705 11697 fs_manager.cc:359] Metadata directory not provided
I20260620 18:41:03.780853 11697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:41:03.781162 11697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:41:03.784235 11697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980527431848-22676-0/minicluster-data/master-0/data/instance:
uuid: "1001b29067eb4878b0b0db9245ada04e"
format_stamp: "Formatted at 2026-06-20 18:41:03 on dist-test-slave-5lgp"
server_key: "7e03852bc0b7c3c4ecedd3ace3cbdde0"
server_key_iv: "85c8ba6bfbdb295522ba345794d43ead"
server_key_version: "encryptionkey@0"
I20260620 18:41:03.784965 11697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980527431848-22676-0/minicluster-data/master-0/wal/instance:
uuid: "1001b29067eb4878b0b0db9245ada04e"
format_stamp: "Formatted at 2026-06-20 18:41:03 on dist-test-slave-5lgp"
server_key: "7e03852bc0b7c3c4ecedd3ace3cbdde0"
server_key_iv: "85c8ba6bfbdb295522ba345794d43ead"
server_key_version: "encryptionkey@0"
I20260620 18:41:03.790364 11697 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260620 18:41:03.793735 11712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:03.795035 11697 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 18:41:03.795230 11697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980527431848-22676-0/minicluster-data/master-0/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980527431848-22676-0/minicluster-data/master-0/wal
uuid: "1001b29067eb4878b0b0db9245ada04e"
format_stamp: "Formatted at 2026-06-20 18:41:03 on dist-test-slave-5lgp"
server_key: "7e03852bc0b7c3c4ecedd3ace3cbdde0"
server_key_iv: "85c8ba6bfbdb295522ba345794d43ead"
server_key_version: "encryptionkey@0"
I20260620 18:41:03.795435 11697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980527431848-22676-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980527431848-22676-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980527431848-22676-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:41:03.822984 11697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:41:03.827221 11697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:41:03.827520 11697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:41:03.845628 11697 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40019
I20260620 18:41:03.845647 11763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40019 every 8 connection(s)
I20260620 18:41:03.847019 11697 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
I20260620 18:41:03.850642 11764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:41:03.855552 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 11697
I20260620 18:41:03.855746 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980527431848-22676-0/minicluster-data/master-0/wal/instance
I20260620 18:41:03.856302 22676 external_mini_cluster.cc:1428] Setting key 5429af01ea9de9eec6c7f986c9e1f7ca
I20260620 18:41:03.861958 11764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1001b29067eb4878b0b0db9245ada04e: Bootstrap starting.
I20260620 18:41:03.865578 11764 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1001b29067eb4878b0b0db9245ada04e: Neither blocks nor log segments found. Creating new log.
I20260620 18:41:03.866827 11764 log.cc:824] T 00000000000000000000000000000000 P 1001b29067eb4878b0b0db9245ada04e: Log is configured to *not* fsync() on all Append() calls
I20260620 18:41:03.870077 11764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1001b29067eb4878b0b0db9245ada04e: No bootstrap required, opened a new log
I20260620 18:41:03.870219 11767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:03.858078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.106:47251 (local address 127.0.0.1:40019)
0620 18:41:03.858866 (+ 788us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:03.858878 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:03.858917 (+ 39us) server_negotiation.cc:407] Connection header received
0620 18:41:03.860647 (+ 1730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:03.860671 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:03.861065 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:03.861422 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:03.861985 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:03.863339 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:03.865246 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:03.866062 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:03.866195 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:03.867649 (+ 1454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:03.867660 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:03.867682 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:41:03.868423 (+ 741us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:03.868494 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:41:03.868530 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:03.868657 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:03.868926 (+ 269us) server_negotiation.cc:299] Negotiation successful
0620 18:41:03.869116 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":357,"threads_started":1}
I20260620 18:41:03.874137 22676 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:41:03.878046 11764 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1001b29067eb4878b0b0db9245ada04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1001b29067eb4878b0b0db9245ada04e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40019 } }
I20260620 18:41:03.878455 11764 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1001b29067eb4878b0b0db9245ada04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:41:03.878598 11764 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1001b29067eb4878b0b0db9245ada04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1001b29067eb4878b0b0db9245ada04e, State: Initialized, Role: FOLLOWER
I20260620 18:41:03.879279 11764 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1001b29067eb4878b0b0db9245ada04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1001b29067eb4878b0b0db9245ada04e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40019 } }
I20260620 18:41:03.879598 11764 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1001b29067eb4878b0b0db9245ada04e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:41:03.879766 11764 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1001b29067eb4878b0b0db9245ada04e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:41:03.879990 11764 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1001b29067eb4878b0b0db9245ada04e [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:41:03.883162 11764 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1001b29067eb4878b0b0db9245ada04e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1001b29067eb4878b0b0db9245ada04e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40019 } }
I20260620 18:41:03.883744 11764 leader_election.cc:302] T 00000000000000000000000000000000 P 1001b29067eb4878b0b0db9245ada04e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1001b29067eb4878b0b0db9245ada04e; no voters:
I20260620 18:41:03.884414 11764 leader_election.cc:288] T 00000000000000000000000000000000 P 1001b29067eb4878b0b0db9245ada04e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:41:03.884554 11774 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1001b29067eb4878b0b0db9245ada04e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:41:03.886407 11767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:03.878706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.106:55777 (local address 127.0.0.1:40019)
0620 18:41:03.878929 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:03.878938 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:03.879397 (+ 459us) server_negotiation.cc:407] Connection header received
0620 18:41:03.879628 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:03.879633 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:03.879723 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:03.879820 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:03.880444 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:03.881224 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:03.883077 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:03.883591 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:03.883718 (+ 127us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:03.885140 (+ 1422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:03.885145 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:03.885149 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:41:03.885173 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:03.885218 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:41:03.885238 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:03.885343 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:03.885834 (+ 491us) server_negotiation.cc:299] Negotiation successful
0620 18:41:03.885917 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 18:41:03.886701 11764 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1001b29067eb4878b0b0db9245ada04e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:41:03.891415 11774 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1001b29067eb4878b0b0db9245ada04e [term 1 LEADER]: Becoming Leader. State: Replica: 1001b29067eb4878b0b0db9245ada04e, State: Running, Role: LEADER
I20260620 18:41:03.892015 11774 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1001b29067eb4878b0b0db9245ada04e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1001b29067eb4878b0b0db9245ada04e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40019 } }
I20260620 18:41:03.895339 11775 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1001b29067eb4878b0b0db9245ada04e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1001b29067eb4878b0b0db9245ada04e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1001b29067eb4878b0b0db9245ada04e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40019 } } }
I20260620 18:41:03.895761 11775 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1001b29067eb4878b0b0db9245ada04e [sys.catalog]: This master's current role is: LEADER
I20260620 18:41:03.895646 11776 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1001b29067eb4878b0b0db9245ada04e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1001b29067eb4878b0b0db9245ada04e. Latest consensus state: current_term: 1 leader_uuid: "1001b29067eb4878b0b0db9245ada04e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1001b29067eb4878b0b0db9245ada04e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40019 } } }
I20260620 18:41:03.895887 11776 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1001b29067eb4878b0b0db9245ada04e [sys.catalog]: This master's current role is: LEADER
I20260620 18:41:03.898748 11789 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:41:03.903983 11789 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:41:03.912617 11789 catalog_manager.cc:1347] Generated new cluster ID: b6a707b5565f4518ad46060ef4491659
I20260620 18:41:03.912742 11789 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:41:03.933815 11789 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:41:03.935639 11789 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:41:03.942971 11789 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1001b29067eb4878b0b0db9245ada04e: Generated new TSK 0
I20260620 18:41:03.943976 11789 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:41:03.957731 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 11697
W20260620 18:41:03.958946 11759 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.106:47251
W20260620 18:41:03.959764 22676 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 11697: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:41:03.960135 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 11697
W20260620 18:41:03.980587 11682 connection.cc:535] client connection to 127.0.0.1:40019 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40019): Connection reset by peer (error 104)
2026-06-20T18:41:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1931 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980527431848-22676-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:41:04 dist-test-slave-5lgp krb5kdc[11795](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:41:04 dist-test-slave-5lgp krb5kdc[11795](info): set up 2 sockets
Jun 20 18:41:04 dist-test-slave-5lgp krb5kdc[11795](info): commencing operation
krb5kdc: starting...
W20260620 18:41:05.062225 22676 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980527431848-22676-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980527431848-22676-0/krb5kdc/test-user.keytab.
Jun 20 18:41:05 dist-test-slave-5lgp krb5kdc[11795](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980865, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:41:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:41:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980527431848-22676-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980527431848-22676-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 18:41:05.310632 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980527431848-22676-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980527431848-22676-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980527431848-22676-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34423
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980527431848-22676-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34423
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:41:05.635898 11808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:41:05.636196 11808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:41:05.636291 11808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:41:05.643761 11808 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:41:05.643888 11808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:41:05.643937 11808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:41:05.644007 11808 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:41:05.644052 11808 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:41:05.654109 11808 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980527431848-22676-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980527431848-22676-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34423
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980527431848-22676-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34423
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980527431848-22676-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:41:05.655355 11808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:41:05.656615 11808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:41:05.668705 11813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:05.668790 11816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:05.668732 11814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:41:05.669868 11808 server_base.cc:1034] running on GCE node
I20260620 18:41:05.670730 11808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:41:05.672092 11808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:41:05.673292 11808 hybrid_clock.cc:648] HybridClock initialized: now 1781980865673252 us; error 30 us; skew 500 ppm
I20260620 18:41:05.673699 11808 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:41:05 dist-test-slave-5lgp krb5kdc[11795](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980865, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:41:05.676877 11808 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 18:41:05.678350 11808 webserver.cc:460] Webserver started at http://127.0.0.1:41503/ using document root <none> and password file <none>
I20260620 18:41:05.679049 11808 fs_manager.cc:359] Metadata directory not provided
I20260620 18:41:05.679189 11808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:41:05.679502 11808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:41:05.682611 11808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980527431848-22676-0/minicluster-data/master-0/data/instance:
uuid: "02ae28f4d3bb4befa568c788f2d6065d"
format_stamp: "Formatted at 2026-06-20 18:41:05 on dist-test-slave-5lgp"
server_key: "4f37b3468761e81d4cce7d519eb4b6c0"
server_key_iv: "bacc4d84b25790e1d510876b47e2313f"
server_key_version: "encryptionkey@0"
I20260620 18:41:05.683378 11808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980527431848-22676-0/minicluster-data/master-0/wal/instance:
uuid: "02ae28f4d3bb4befa568c788f2d6065d"
format_stamp: "Formatted at 2026-06-20 18:41:05 on dist-test-slave-5lgp"
server_key: "4f37b3468761e81d4cce7d519eb4b6c0"
server_key_iv: "bacc4d84b25790e1d510876b47e2313f"
server_key_version: "encryptionkey@0"
I20260620 18:41:05.688833 11808 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 18:41:05.692378 11823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:05.693840 11808 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:41:05.694025 11808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980527431848-22676-0/minicluster-data/master-0/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980527431848-22676-0/minicluster-data/master-0/wal
uuid: "02ae28f4d3bb4befa568c788f2d6065d"
format_stamp: "Formatted at 2026-06-20 18:41:05 on dist-test-slave-5lgp"
server_key: "4f37b3468761e81d4cce7d519eb4b6c0"
server_key_iv: "bacc4d84b25790e1d510876b47e2313f"
server_key_version: "encryptionkey@0"
I20260620 18:41:05.694227 11808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980527431848-22676-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980527431848-22676-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980527431848-22676-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:41:05.699920 11808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:41:05.704162 11808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:41:05.704447 11808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:41:05.722150 11808 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34423
I20260620 18:41:05.722195 11874 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34423 every 8 connection(s)
I20260620 18:41:05.723546 11808 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
I20260620 18:41:05.724670 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 11808
I20260620 18:41:05.724859 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980527431848-22676-0/minicluster-data/master-0/wal/instance
I20260620 18:41:05.725296 22676 external_mini_cluster.cc:1428] Setting key 651d996cad4bc23766e4577bb49e9cea
I20260620 18:41:05.728039 11875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:41:05.734256 11878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:05.727227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.106:46257 (local address 127.0.0.1:34423)
0620 18:41:05.727989 (+ 762us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:05.728007 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:05.728065 (+ 58us) server_negotiation.cc:407] Connection header received
0620 18:41:05.730380 (+ 2315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:05.730407 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:05.730843 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:05.731311 (+ 468us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:05.732293 (+ 982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:05.732301 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:05.732317 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:41:05.732339 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:05.732439 (+ 100us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:41:05.732475 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:05.732685 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:05.732817 (+ 132us) server_negotiation.cc:299] Negotiation successful
0620 18:41:05.733012 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":313,"threads_started":1}
I20260620 18:41:05.738176 22676 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:41:05.741024 11875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02ae28f4d3bb4befa568c788f2d6065d: Bootstrap starting.
I20260620 18:41:05.744481 11875 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 02ae28f4d3bb4befa568c788f2d6065d: Neither blocks nor log segments found. Creating new log.
I20260620 18:41:05.744868 11878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:05.742591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.106:34735 (local address 127.0.0.1:34423)
0620 18:41:05.742770 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:05.742777 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:05.743058 (+ 281us) server_negotiation.cc:407] Connection header received
0620 18:41:05.743184 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:05.743189 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:05.743243 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:05.743340 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:05.744232 (+ 892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:05.744239 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:05.744247 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:41:05.744269 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:05.744317 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:41:05.744341 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:05.744484 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:05.744531 (+ 47us) server_negotiation.cc:299] Negotiation successful
0620 18:41:05.744591 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 18:41:05.745734 11875 log.cc:824] T 00000000000000000000000000000000 P 02ae28f4d3bb4befa568c788f2d6065d: Log is configured to *not* fsync() on all Append() calls
I20260620 18:41:05.748338 11875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02ae28f4d3bb4befa568c788f2d6065d: No bootstrap required, opened a new log
I20260620 18:41:05.755295 11875 raft_consensus.cc:348] T 00000000000000000000000000000000 P 02ae28f4d3bb4befa568c788f2d6065d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02ae28f4d3bb4befa568c788f2d6065d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34423 } }
I20260620 18:41:05.755663 11875 raft_consensus.cc:374] T 00000000000000000000000000000000 P 02ae28f4d3bb4befa568c788f2d6065d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:41:05.755764 11875 raft_consensus.cc:729] T 00000000000000000000000000000000 P 02ae28f4d3bb4befa568c788f2d6065d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02ae28f4d3bb4befa568c788f2d6065d, State: Initialized, Role: FOLLOWER
I20260620 18:41:05.756315 11875 consensus_queue.cc:260] T 00000000000000000000000000000000 P 02ae28f4d3bb4befa568c788f2d6065d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02ae28f4d3bb4befa568c788f2d6065d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34423 } }
I20260620 18:41:05.756562 11875 raft_consensus.cc:388] T 00000000000000000000000000000000 P 02ae28f4d3bb4befa568c788f2d6065d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:41:05.756708 11875 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02ae28f4d3bb4befa568c788f2d6065d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:41:05.756883 11875 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 02ae28f4d3bb4befa568c788f2d6065d [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:41:05.759035 11875 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02ae28f4d3bb4befa568c788f2d6065d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02ae28f4d3bb4befa568c788f2d6065d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34423 } }
I20260620 18:41:05.759541 11875 leader_election.cc:302] T 00000000000000000000000000000000 P 02ae28f4d3bb4befa568c788f2d6065d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 02ae28f4d3bb4befa568c788f2d6065d; no voters:
I20260620 18:41:05.760114 11875 leader_election.cc:288] T 00000000000000000000000000000000 P 02ae28f4d3bb4befa568c788f2d6065d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:41:05.760288 11885 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 02ae28f4d3bb4befa568c788f2d6065d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:41:05.761106 11885 raft_consensus.cc:686] T 00000000000000000000000000000000 P 02ae28f4d3bb4befa568c788f2d6065d [term 1 LEADER]: Becoming Leader. State: Replica: 02ae28f4d3bb4befa568c788f2d6065d, State: Running, Role: LEADER
I20260620 18:41:05.761662 11885 consensus_queue.cc:237] T 00000000000000000000000000000000 P 02ae28f4d3bb4befa568c788f2d6065d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02ae28f4d3bb4befa568c788f2d6065d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34423 } }
I20260620 18:41:05.762245 11875 sys_catalog.cc:564] T 00000000000000000000000000000000 P 02ae28f4d3bb4befa568c788f2d6065d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:41:05.764604 11886 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02ae28f4d3bb4befa568c788f2d6065d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "02ae28f4d3bb4befa568c788f2d6065d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02ae28f4d3bb4befa568c788f2d6065d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34423 } } }
I20260620 18:41:05.765128 11886 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02ae28f4d3bb4befa568c788f2d6065d [sys.catalog]: This master's current role is: LEADER
I20260620 18:41:05.766852 11887 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02ae28f4d3bb4befa568c788f2d6065d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02ae28f4d3bb4befa568c788f2d6065d. Latest consensus state: current_term: 1 leader_uuid: "02ae28f4d3bb4befa568c788f2d6065d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02ae28f4d3bb4befa568c788f2d6065d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34423 } } }
I20260620 18:41:05.767283 11887 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02ae28f4d3bb4befa568c788f2d6065d [sys.catalog]: This master's current role is: LEADER
I20260620 18:41:05.768519 11894 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:41:05.773765 11894 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:41:05.782958 11894 catalog_manager.cc:1347] Generated new cluster ID: 1d675705c26f446a9b4124e09192193b
I20260620 18:41:05.783084 11894 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:41:05.792423 11894 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:41:05.792838 11894 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:41:05.809072 11894 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 02ae28f4d3bb4befa568c788f2d6065d: Generated new TSK 0
I20260620 18:41:05.809964 11894 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:41:05.820926 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 11808
W20260620 18:41:05.822093 11870 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.106:46257
W20260620 18:41:05.822810 22676 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 11808: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:41:05.822988 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 11808
2026-06-20T18:41:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1858 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980527431848-22676-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:41:05 dist-test-slave-5lgp krb5kdc[11906](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:41:05 dist-test-slave-5lgp krb5kdc[11906](info): set up 2 sockets
Jun 20 18:41:05 dist-test-slave-5lgp krb5kdc[11906](info): commencing operation
krb5kdc: starting...
W20260620 18:41:06.917737 22676 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980527431848-22676-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980527431848-22676-0/krb5kdc/test-user.keytab.
Jun 20 18:41:07 dist-test-slave-5lgp krb5kdc[11906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980867, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:41:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:41:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.37.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.37.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.37.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab.
Entry for principal kudu/127.22.37.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab.
I20260620 18:41:07.168388 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980527431848-22676-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980527431848-22676-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980527431848-22676-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.62:37539
--webserver_interface=127.22.37.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab
--principal=kudu/127.22.37.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:34683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.37.62:37539
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:41:07.499029 11919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:41:07.499331 11919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:41:07.499529 11919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:41:07.506848 11919 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:41:07.506961 11919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:41:07.507072 11919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:41:07.507174 11919 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:41:07.507252 11919 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:41:07.517359 11919 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:34683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980527431848-22676-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980527431848-22676-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.37.62:37539
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab
--principal=kudu/127.22.37.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.62:37539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.37.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980527431848-22676-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:41:07.518775 11919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:41:07.520136 11919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:41:07.532171 11924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:07.532207 11925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:07.532135 11927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:41:07.533221 11919 server_base.cc:1034] running on GCE node
I20260620 18:41:07.534116 11919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:41:07.535573 11919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:41:07.536803 11919 hybrid_clock.cc:648] HybridClock initialized: now 1781980867536765 us; error 54 us; skew 500 ppm
I20260620 18:41:07.537161 11919 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:41:07 dist-test-slave-5lgp krb5kdc[11906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980867, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:41:07.540351 11919 init.cc:377] Logged in from keytab as kudu/127.22.37.62@KRBTEST.COM (short username kudu)
I20260620 18:41:07.541855 11919 webserver.cc:460] Webserver started at http://127.22.37.62:38925/ using document root <none> and password file <none>
I20260620 18:41:07.542527 11919 fs_manager.cc:359] Metadata directory not provided
I20260620 18:41:07.542629 11919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:41:07.542903 11919 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:41:07.545917 11919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980527431848-22676-0/minicluster-data/master-0/data/instance:
uuid: "e2a90cc132844ca082359e3776b05085"
format_stamp: "Formatted at 2026-06-20 18:41:07 on dist-test-slave-5lgp"
server_key: "ef4bf8c2b31d88c92c3609090c1575a0"
server_key_iv: "18b6d9be74160d13d9c36884eecffc83"
server_key_version: "encryptionkey@0"
I20260620 18:41:07.546623 11919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980527431848-22676-0/minicluster-data/master-0/wal/instance:
uuid: "e2a90cc132844ca082359e3776b05085"
format_stamp: "Formatted at 2026-06-20 18:41:07 on dist-test-slave-5lgp"
server_key: "ef4bf8c2b31d88c92c3609090c1575a0"
server_key_iv: "18b6d9be74160d13d9c36884eecffc83"
server_key_version: "encryptionkey@0"
I20260620 18:41:07.551821 11919 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:41:07.555378 11934 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:07.556725 11919 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:41:07.556859 11919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980527431848-22676-0/minicluster-data/master-0/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980527431848-22676-0/minicluster-data/master-0/wal
uuid: "e2a90cc132844ca082359e3776b05085"
format_stamp: "Formatted at 2026-06-20 18:41:07 on dist-test-slave-5lgp"
server_key: "ef4bf8c2b31d88c92c3609090c1575a0"
server_key_iv: "18b6d9be74160d13d9c36884eecffc83"
server_key_version: "encryptionkey@0"
I20260620 18:41:07.557022 11919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980527431848-22676-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980527431848-22676-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980527431848-22676-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:41:07.591509 11919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:41:07.595870 11919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:41:07.596202 11919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:41:07.616190 11919 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.62:37539
I20260620 18:41:07.616209 11985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.62:37539 every 8 connection(s)
I20260620 18:41:07.617832 11919 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
I20260620 18:41:07.621706 11986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:41:07.623136 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 11919
I20260620 18:41:07.623318 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980527431848-22676-0/minicluster-data/master-0/wal/instance
I20260620 18:41:07.623829 22676 external_mini_cluster.cc:1428] Setting key c561d2e89937a2e3061c2323263f5f8a
I20260620 18:41:07.636351 11986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2a90cc132844ca082359e3776b05085: Bootstrap starting.
Jun 20 18:41:07 dist-test-slave-5lgp krb5kdc[11906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980867, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM
I20260620 18:41:07.639554 11986 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e2a90cc132844ca082359e3776b05085: Neither blocks nor log segments found. Creating new log.
I20260620 18:41:07.640982 11986 log.cc:824] T 00000000000000000000000000000000 P e2a90cc132844ca082359e3776b05085: Log is configured to *not* fsync() on all Append() calls
I20260620 18:41:07.644464 11986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2a90cc132844ca082359e3776b05085: No bootstrap required, opened a new log
I20260620 18:41:07.648378 11989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:07.625922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57655 (local address 127.22.37.62:37539)
0620 18:41:07.626761 (+ 839us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:07.626783 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:07.626839 (+ 56us) server_negotiation.cc:407] Connection header received
0620 18:41:07.628518 (+ 1679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:07.628545 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:07.629089 (+ 544us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:07.629578 (+ 489us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:07.630875 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:07.632860 (+ 1985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:07.634213 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:07.635264 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:07.635466 (+ 202us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:07.639553 (+ 4087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:07.639564 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:07.639583 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:07.639636 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:07.643403 (+ 3767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:07.644215 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:07.644222 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:07.644230 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:07.644429 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:07.644945 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:07.644951 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:07.644954 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:07.645707 (+ 753us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:07.646040 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:07.646628 (+ 588us) server_negotiation.cc:299] Negotiation successful
0620 18:41:07.646984 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":340,"threads_started":1}
I20260620 18:41:07.652505 22676 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:41:07.654136 11986 raft_consensus.cc:348] T 00000000000000000000000000000000 P e2a90cc132844ca082359e3776b05085 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2a90cc132844ca082359e3776b05085" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 37539 } }
I20260620 18:41:07.654507 11986 raft_consensus.cc:374] T 00000000000000000000000000000000 P e2a90cc132844ca082359e3776b05085 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:41:07.654604 11986 raft_consensus.cc:729] T 00000000000000000000000000000000 P e2a90cc132844ca082359e3776b05085 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2a90cc132844ca082359e3776b05085, State: Initialized, Role: FOLLOWER
I20260620 18:41:07.655280 11986 consensus_queue.cc:260] T 00000000000000000000000000000000 P e2a90cc132844ca082359e3776b05085 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2a90cc132844ca082359e3776b05085" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 37539 } }
I20260620 18:41:07.655551 11986 raft_consensus.cc:388] T 00000000000000000000000000000000 P e2a90cc132844ca082359e3776b05085 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:41:07.655705 11986 raft_consensus.cc:482] T 00000000000000000000000000000000 P e2a90cc132844ca082359e3776b05085 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:41:07.655889 11986 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e2a90cc132844ca082359e3776b05085 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:41:07.658609 11986 raft_consensus.cc:504] T 00000000000000000000000000000000 P e2a90cc132844ca082359e3776b05085 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2a90cc132844ca082359e3776b05085" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 37539 } }
I20260620 18:41:07.659109 11986 leader_election.cc:302] T 00000000000000000000000000000000 P e2a90cc132844ca082359e3776b05085 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e2a90cc132844ca082359e3776b05085; no voters:
I20260620 18:41:07.659755 11986 leader_election.cc:288] T 00000000000000000000000000000000 P e2a90cc132844ca082359e3776b05085 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:41:07.659912 11996 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e2a90cc132844ca082359e3776b05085 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:41:07.662578 11986 sys_catalog.cc:564] T 00000000000000000000000000000000 P e2a90cc132844ca082359e3776b05085 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:41:07.666675 11996 raft_consensus.cc:686] T 00000000000000000000000000000000 P e2a90cc132844ca082359e3776b05085 [term 1 LEADER]: Becoming Leader. State: Replica: e2a90cc132844ca082359e3776b05085, State: Running, Role: LEADER
I20260620 18:41:07.667398 11996 consensus_queue.cc:237] T 00000000000000000000000000000000 P e2a90cc132844ca082359e3776b05085 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2a90cc132844ca082359e3776b05085" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 37539 } }
I20260620 18:41:07.670871 11997 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2a90cc132844ca082359e3776b05085 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e2a90cc132844ca082359e3776b05085" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2a90cc132844ca082359e3776b05085" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 37539 } } }
I20260620 18:41:07.671339 11997 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2a90cc132844ca082359e3776b05085 [sys.catalog]: This master's current role is: LEADER
I20260620 18:41:07.673821 11989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:07.656961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33057 (local address 127.22.37.62:37539)
0620 18:41:07.657138 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:07.657142 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:07.657714 (+ 572us) server_negotiation.cc:407] Connection header received
0620 18:41:07.657912 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:07.657917 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:07.658010 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:07.658112 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:07.659689 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:07.660945 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:07.662497 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:07.663050 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:07.663188 (+ 138us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:07.666036 (+ 2848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:07.666041 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:07.666044 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:07.666076 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:07.669017 (+ 2941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:07.670024 (+ 1007us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:07.670031 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:07.670034 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:07.670173 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:07.670804 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:07.670808 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:07.670811 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:07.671273 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:07.671555 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:07.673031 (+ 1476us) server_negotiation.cc:299] Negotiation successful
0620 18:41:07.673282 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 18:41:07.677906 11998 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2a90cc132844ca082359e3776b05085 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e2a90cc132844ca082359e3776b05085. Latest consensus state: current_term: 1 leader_uuid: "e2a90cc132844ca082359e3776b05085" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2a90cc132844ca082359e3776b05085" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 37539 } } }
I20260620 18:41:07.678411 11998 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2a90cc132844ca082359e3776b05085 [sys.catalog]: This master's current role is: LEADER
I20260620 18:41:07.681958 12004 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:41:07.686486 12004 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:41:07.695528 12004 catalog_manager.cc:1347] Generated new cluster ID: 2910ea3454954626aa50f22d1070ed34
I20260620 18:41:07.696017 12004 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:41:07.719012 12004 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:41:07.720400 12004 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:41:07.729041 12004 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e2a90cc132844ca082359e3776b05085: Generated new TSK 0
I20260620 18:41:07.730180 12004 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:41:07.743835 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 11919
I20260620 18:41:07.745307 11981 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:41:07.835762 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 11919
W20260620 18:41:07.852792 11904 connection.cc:535] client connection to 127.22.37.62:37539 recv error: Network error: failed to read from TLS socket (remote: 127.22.37.62:37539): Connection reset by peer (error 104)
2026-06-20T18:41:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2013 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980527431848-22676-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:41:07 dist-test-slave-5lgp krb5kdc[12019](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:41:07 dist-test-slave-5lgp krb5kdc[12019](info): set up 2 sockets
Jun 20 18:41:07 dist-test-slave-5lgp krb5kdc[12019](info): commencing operation
krb5kdc: starting...
W20260620 18:41:08.945952 22676 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980527431848-22676-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980527431848-22676-0/krb5kdc/test-user.keytab.
Jun 20 18:41:09 dist-test-slave-5lgp krb5kdc[12019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980869, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:41:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:41:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.37.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.37.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.37.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab.
Entry for principal kudu/127.22.37.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab.
I20260620 18:41:09.204651 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980527431848-22676-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980527431848-22676-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980527431848-22676-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.62:43649
--webserver_interface=127.22.37.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab
--principal=kudu/127.22.37.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:33519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.37.62:43649
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:41:09.534228 12032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:41:09.534550 12032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:41:09.534755 12032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:41:09.542080 12032 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:41:09.542220 12032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:41:09.542306 12032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:41:09.542423 12032 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:41:09.542480 12032 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:41:09.552384 12032 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:33519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980527431848-22676-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980527431848-22676-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.37.62:43649
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab
--principal=kudu/127.22.37.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.62:43649
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.37.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980527431848-22676-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:41:09.553817 12032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:41:09.555159 12032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:41:09.567107 12037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:09.567062 12040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:09.567147 12038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:41:09.568341 12032 server_base.cc:1034] running on GCE node
I20260620 18:41:09.569106 12032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:41:09.570559 12032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:41:09.571825 12032 hybrid_clock.cc:648] HybridClock initialized: now 1781980869571748 us; error 60 us; skew 500 ppm
I20260620 18:41:09.572181 12032 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:41:09 dist-test-slave-5lgp krb5kdc[12019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980869, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:41:09.578351 12032 init.cc:377] Logged in from keytab as kudu/127.22.37.62@KRBTEST.COM (short username kudu)
I20260620 18:41:09.579873 12032 webserver.cc:460] Webserver started at http://127.22.37.62:44391/ using document root <none> and password file <none>
I20260620 18:41:09.580531 12032 fs_manager.cc:359] Metadata directory not provided
I20260620 18:41:09.580708 12032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:41:09.581019 12032 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:41:09.584100 12032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980527431848-22676-0/minicluster-data/master-0/data/instance:
uuid: "803ab467a50d4bd2a9a58c4110414d93"
format_stamp: "Formatted at 2026-06-20 18:41:09 on dist-test-slave-5lgp"
server_key: "e04fdeb193f68f3ef40b3f07eca087c6"
server_key_iv: "77f5ebaf8599442330f5c7630982ff7e"
server_key_version: "encryptionkey@0"
I20260620 18:41:09.584858 12032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980527431848-22676-0/minicluster-data/master-0/wal/instance:
uuid: "803ab467a50d4bd2a9a58c4110414d93"
format_stamp: "Formatted at 2026-06-20 18:41:09 on dist-test-slave-5lgp"
server_key: "e04fdeb193f68f3ef40b3f07eca087c6"
server_key_iv: "77f5ebaf8599442330f5c7630982ff7e"
server_key_version: "encryptionkey@0"
I20260620 18:41:09.589898 12032 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260620 18:41:09.593516 12047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:09.594862 12032 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:41:09.595072 12032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980527431848-22676-0/minicluster-data/master-0/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980527431848-22676-0/minicluster-data/master-0/wal
uuid: "803ab467a50d4bd2a9a58c4110414d93"
format_stamp: "Formatted at 2026-06-20 18:41:09 on dist-test-slave-5lgp"
server_key: "e04fdeb193f68f3ef40b3f07eca087c6"
server_key_iv: "77f5ebaf8599442330f5c7630982ff7e"
server_key_version: "encryptionkey@0"
I20260620 18:41:09.595312 12032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980527431848-22676-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980527431848-22676-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980527431848-22676-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:41:09.612883 12032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:41:09.617033 12032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:41:09.617275 12032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:41:09.638913 12032 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.62:43649
I20260620 18:41:09.638967 12098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.62:43649 every 8 connection(s)
I20260620 18:41:09.641206 12032 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
I20260620 18:41:09.646828 12099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:41:09.649751 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 12032
I20260620 18:41:09.649952 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980527431848-22676-0/minicluster-data/master-0/wal/instance
I20260620 18:41:09.650607 22676 external_mini_cluster.cc:1428] Setting key ca65f49bb9dca514de21152dc68aadec
I20260620 18:41:09.663475 12099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 803ab467a50d4bd2a9a58c4110414d93: Bootstrap starting.
I20260620 18:41:09.667994 12099 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 803ab467a50d4bd2a9a58c4110414d93: Neither blocks nor log segments found. Creating new log.
I20260620 18:41:09.668649 12102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:09.653041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46567 (local address 127.22.37.62:43649)
0620 18:41:09.654005 (+ 964us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:09.654020 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:09.654075 (+ 55us) server_negotiation.cc:407] Connection header received
0620 18:41:09.656629 (+ 2554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:09.656653 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:09.657073 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:09.657515 (+ 442us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:09.658223 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:09.659704 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:09.662146 (+ 2442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:09.663244 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:09.663446 (+ 202us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:09.665584 (+ 2138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:09.665598 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:09.665616 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:41:09.666386 (+ 770us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:09.666488 (+ 102us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:41:09.666523 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:09.666709 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:09.667022 (+ 313us) server_negotiation.cc:299] Negotiation successful
0620 18:41:09.667197 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":679,"thread_start_us":433,"threads_started":1}
I20260620 18:41:09.669543 12099 log.cc:824] T 00000000000000000000000000000000 P 803ab467a50d4bd2a9a58c4110414d93: Log is configured to *not* fsync() on all Append() calls
I20260620 18:41:09.673779 22676 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:41:09.676457 12099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 803ab467a50d4bd2a9a58c4110414d93: No bootstrap required, opened a new log
I20260620 18:41:09.686239 12099 raft_consensus.cc:348] T 00000000000000000000000000000000 P 803ab467a50d4bd2a9a58c4110414d93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "803ab467a50d4bd2a9a58c4110414d93" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 43649 } }
I20260620 18:41:09.686681 12099 raft_consensus.cc:374] T 00000000000000000000000000000000 P 803ab467a50d4bd2a9a58c4110414d93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:41:09.686803 12099 raft_consensus.cc:729] T 00000000000000000000000000000000 P 803ab467a50d4bd2a9a58c4110414d93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 803ab467a50d4bd2a9a58c4110414d93, State: Initialized, Role: FOLLOWER
I20260620 18:41:09.687461 12099 consensus_queue.cc:260] T 00000000000000000000000000000000 P 803ab467a50d4bd2a9a58c4110414d93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "803ab467a50d4bd2a9a58c4110414d93" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 43649 } }
I20260620 18:41:09.687791 12099 raft_consensus.cc:388] T 00000000000000000000000000000000 P 803ab467a50d4bd2a9a58c4110414d93 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:41:09.687913 12102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:09.679063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40745 (local address 127.22.37.62:43649)
0620 18:41:09.679248 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:09.679254 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:09.679762 (+ 508us) server_negotiation.cc:407] Connection header received
0620 18:41:09.679989 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:09.679995 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:09.680085 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:09.680192 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:09.680889 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:09.682079 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:09.684240 (+ 2161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:09.685006 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:09.685137 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:09.686890 (+ 1753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:09.686898 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:09.686904 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:41:09.686930 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:09.687014 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:41:09.687044 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:09.687192 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:09.687423 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 18:41:09.687486 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 18:41:09.687963 12099 raft_consensus.cc:482] T 00000000000000000000000000000000 P 803ab467a50d4bd2a9a58c4110414d93 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:41:09.688161 12099 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 803ab467a50d4bd2a9a58c4110414d93 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:41:09.690703 12099 raft_consensus.cc:504] T 00000000000000000000000000000000 P 803ab467a50d4bd2a9a58c4110414d93 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "803ab467a50d4bd2a9a58c4110414d93" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 43649 } }
I20260620 18:41:09.691151 12099 leader_election.cc:302] T 00000000000000000000000000000000 P 803ab467a50d4bd2a9a58c4110414d93 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 803ab467a50d4bd2a9a58c4110414d93; no voters:
I20260620 18:41:09.691651 12099 leader_election.cc:288] T 00000000000000000000000000000000 P 803ab467a50d4bd2a9a58c4110414d93 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:41:09.691800 12109 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 803ab467a50d4bd2a9a58c4110414d93 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:41:09.692528 12109 raft_consensus.cc:686] T 00000000000000000000000000000000 P 803ab467a50d4bd2a9a58c4110414d93 [term 1 LEADER]: Becoming Leader. State: Replica: 803ab467a50d4bd2a9a58c4110414d93, State: Running, Role: LEADER
I20260620 18:41:09.693099 12109 consensus_queue.cc:237] T 00000000000000000000000000000000 P 803ab467a50d4bd2a9a58c4110414d93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "803ab467a50d4bd2a9a58c4110414d93" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 43649 } }
I20260620 18:41:09.693723 12099 sys_catalog.cc:564] T 00000000000000000000000000000000 P 803ab467a50d4bd2a9a58c4110414d93 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:41:09.696738 12111 sys_catalog.cc:455] T 00000000000000000000000000000000 P 803ab467a50d4bd2a9a58c4110414d93 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 803ab467a50d4bd2a9a58c4110414d93. Latest consensus state: current_term: 1 leader_uuid: "803ab467a50d4bd2a9a58c4110414d93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "803ab467a50d4bd2a9a58c4110414d93" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 43649 } } }
I20260620 18:41:09.697242 12111 sys_catalog.cc:458] T 00000000000000000000000000000000 P 803ab467a50d4bd2a9a58c4110414d93 [sys.catalog]: This master's current role is: LEADER
I20260620 18:41:09.696727 12110 sys_catalog.cc:455] T 00000000000000000000000000000000 P 803ab467a50d4bd2a9a58c4110414d93 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "803ab467a50d4bd2a9a58c4110414d93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "803ab467a50d4bd2a9a58c4110414d93" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 43649 } } }
I20260620 18:41:09.698056 12115 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:41:09.698118 12110 sys_catalog.cc:458] T 00000000000000000000000000000000 P 803ab467a50d4bd2a9a58c4110414d93 [sys.catalog]: This master's current role is: LEADER
I20260620 18:41:09.703177 12115 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:41:09.711759 12115 catalog_manager.cc:1347] Generated new cluster ID: 9f53044d8b574c0690e1b9ec251ee8e2
I20260620 18:41:09.711874 12115 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:41:09.740526 12115 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:41:09.741962 12115 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:41:09.752705 12115 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 803ab467a50d4bd2a9a58c4110414d93: Generated new TSK 0
I20260620 18:41:09.753568 12115 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:41:09.768879 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 12032
W20260620 18:41:09.770051 12094 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:46567
W20260620 18:41:09.770792 22676 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 12032: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:41:09.771037 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 12032
W20260620 18:41:09.788522 12017 connection.cc:535] client connection to 127.22.37.62:43649 recv error: Network error: failed to read from TLS socket (remote: 127.22.37.62:43649): Connection reset by peer (error 104)
2026-06-20T18:41:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1935 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980527431848-22676-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:41:09 dist-test-slave-5lgp krb5kdc[12130](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:41:09 dist-test-slave-5lgp krb5kdc[12130](info): set up 2 sockets
Jun 20 18:41:09 dist-test-slave-5lgp krb5kdc[12130](info): commencing operation
krb5kdc: starting...
W20260620 18:41:10.869918 22676 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.005s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980527431848-22676-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980527431848-22676-0/krb5kdc/test-user.keytab.
Jun 20 18:41:11 dist-test-slave-5lgp krb5kdc[12130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980871, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:41:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:41:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.37.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.37.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.37.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab.
Entry for principal kudu/127.22.37.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab.
I20260620 18:41:11.117131 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980527431848-22676-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980527431848-22676-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980527431848-22676-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.62:33745
--webserver_interface=127.22.37.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab
--principal=kudu/127.22.37.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:35927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.37.62:33745
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:41:11.442096 12143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:41:11.442392 12143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:41:11.442564 12143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:41:11.449862 12143 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:41:11.449977 12143 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:41:11.450075 12143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:41:11.450186 12143 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:41:11.450253 12143 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:41:11.460119 12143 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:35927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980527431848-22676-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980527431848-22676-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.37.62:33745
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab
--principal=kudu/127.22.37.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.62:33745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.37.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980527431848-22676-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:41:11.461521 12143 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:41:11.462888 12143 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:41:11.474759 12151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:11.474874 12148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:11.474848 12149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:41:11.476259 12143 server_base.cc:1034] running on GCE node
I20260620 18:41:11.477100 12143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:41:11.478610 12143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:41:11.479820 12143 hybrid_clock.cc:648] HybridClock initialized: now 1781980871479751 us; error 66 us; skew 500 ppm
I20260620 18:41:11.480202 12143 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:41:11 dist-test-slave-5lgp krb5kdc[12130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980871, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:41:11.483373 12143 init.cc:377] Logged in from keytab as kudu/127.22.37.62@KRBTEST.COM (short username kudu)
I20260620 18:41:11.484838 12143 webserver.cc:460] Webserver started at http://127.22.37.62:43309/ using document root <none> and password file <none>
I20260620 18:41:11.485538 12143 fs_manager.cc:359] Metadata directory not provided
I20260620 18:41:11.485641 12143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:41:11.485896 12143 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:41:11.488732 12143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980527431848-22676-0/minicluster-data/master-0/data/instance:
uuid: "a8b8bba4de1e4dc8b6efc2878e222ea9"
format_stamp: "Formatted at 2026-06-20 18:41:11 on dist-test-slave-5lgp"
server_key: "e1c253241a985ba499c8e1c793518478"
server_key_iv: "16f6df50b764109113dbaa6093bd3cff"
server_key_version: "encryptionkey@0"
I20260620 18:41:11.489545 12143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980527431848-22676-0/minicluster-data/master-0/wal/instance:
uuid: "a8b8bba4de1e4dc8b6efc2878e222ea9"
format_stamp: "Formatted at 2026-06-20 18:41:11 on dist-test-slave-5lgp"
server_key: "e1c253241a985ba499c8e1c793518478"
server_key_iv: "16f6df50b764109113dbaa6093bd3cff"
server_key_version: "encryptionkey@0"
I20260620 18:41:11.494524 12143 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 18:41:11.498023 12158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:11.499415 12143 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:41:11.499617 12143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980527431848-22676-0/minicluster-data/master-0/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980527431848-22676-0/minicluster-data/master-0/wal
uuid: "a8b8bba4de1e4dc8b6efc2878e222ea9"
format_stamp: "Formatted at 2026-06-20 18:41:11 on dist-test-slave-5lgp"
server_key: "e1c253241a985ba499c8e1c793518478"
server_key_iv: "16f6df50b764109113dbaa6093bd3cff"
server_key_version: "encryptionkey@0"
I20260620 18:41:11.499816 12143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980527431848-22676-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980527431848-22676-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980527431848-22676-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:41:11.505712 12143 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:41:11.509708 12143 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:41:11.509995 12143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:41:11.528103 12143 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.62:33745
I20260620 18:41:11.528148 12209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.62:33745 every 8 connection(s)
I20260620 18:41:11.529515 12143 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
I20260620 18:41:11.529932 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 12143
I20260620 18:41:11.530109 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980527431848-22676-0/minicluster-data/master-0/wal/instance
I20260620 18:41:11.530725 22676 external_mini_cluster.cc:1428] Setting key cbe8790e30b2718eb3e2cbedb97bae52
I20260620 18:41:11.533933 12210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:41:11.538879 12213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:11.532719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36037 (local address 127.22.37.62:33745)
0620 18:41:11.533625 (+ 906us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:11.533637 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:11.533680 (+ 43us) server_negotiation.cc:407] Connection header received
0620 18:41:11.535408 (+ 1728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:11.535429 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:11.535755 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:11.536118 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:11.537197 (+ 1079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:11.537203 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:11.537216 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:41:11.537230 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:11.537317 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:41:11.537340 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:11.537586 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:11.537701 (+ 115us) server_negotiation.cc:299] Negotiation successful
0620 18:41:11.537903 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":672,"thread_start_us":397,"threads_started":1}
I20260620 18:41:11.542760 22676 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:41:11.546021 12210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8b8bba4de1e4dc8b6efc2878e222ea9: Bootstrap starting.
I20260620 18:41:11.549960 12210 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a8b8bba4de1e4dc8b6efc2878e222ea9: Neither blocks nor log segments found. Creating new log.
I20260620 18:41:11.550670 12213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:11.548129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37483 (local address 127.22.37.62:33745)
0620 18:41:11.548343 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:11.548349 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:11.548671 (+ 322us) server_negotiation.cc:407] Connection header received
0620 18:41:11.548856 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:11.548861 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:11.548905 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:11.548994 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:11.549992 (+ 998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:11.549996 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:11.549999 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:41:11.550013 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:11.550048 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:41:11.550062 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:11.550196 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:11.550339 (+ 143us) server_negotiation.cc:299] Negotiation successful
0620 18:41:11.550384 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 18:41:11.551235 12210 log.cc:824] T 00000000000000000000000000000000 P a8b8bba4de1e4dc8b6efc2878e222ea9: Log is configured to *not* fsync() on all Append() calls
I20260620 18:41:11.553934 12210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8b8bba4de1e4dc8b6efc2878e222ea9: No bootstrap required, opened a new log
I20260620 18:41:11.561003 12210 raft_consensus.cc:348] T 00000000000000000000000000000000 P a8b8bba4de1e4dc8b6efc2878e222ea9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8b8bba4de1e4dc8b6efc2878e222ea9" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 33745 } }
I20260620 18:41:11.561396 12210 raft_consensus.cc:374] T 00000000000000000000000000000000 P a8b8bba4de1e4dc8b6efc2878e222ea9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:41:11.561534 12210 raft_consensus.cc:729] T 00000000000000000000000000000000 P a8b8bba4de1e4dc8b6efc2878e222ea9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8b8bba4de1e4dc8b6efc2878e222ea9, State: Initialized, Role: FOLLOWER
I20260620 18:41:11.562191 12210 consensus_queue.cc:260] T 00000000000000000000000000000000 P a8b8bba4de1e4dc8b6efc2878e222ea9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8b8bba4de1e4dc8b6efc2878e222ea9" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 33745 } }
I20260620 18:41:11.562461 12210 raft_consensus.cc:388] T 00000000000000000000000000000000 P a8b8bba4de1e4dc8b6efc2878e222ea9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:41:11.562609 12210 raft_consensus.cc:482] T 00000000000000000000000000000000 P a8b8bba4de1e4dc8b6efc2878e222ea9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:41:11.562806 12210 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a8b8bba4de1e4dc8b6efc2878e222ea9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:41:11.564796 12210 raft_consensus.cc:504] T 00000000000000000000000000000000 P a8b8bba4de1e4dc8b6efc2878e222ea9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8b8bba4de1e4dc8b6efc2878e222ea9" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 33745 } }
I20260620 18:41:11.565261 12210 leader_election.cc:302] T 00000000000000000000000000000000 P a8b8bba4de1e4dc8b6efc2878e222ea9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a8b8bba4de1e4dc8b6efc2878e222ea9; no voters:
I20260620 18:41:11.565912 12210 leader_election.cc:288] T 00000000000000000000000000000000 P a8b8bba4de1e4dc8b6efc2878e222ea9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:41:11.566040 12220 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a8b8bba4de1e4dc8b6efc2878e222ea9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:41:11.567040 12220 raft_consensus.cc:686] T 00000000000000000000000000000000 P a8b8bba4de1e4dc8b6efc2878e222ea9 [term 1 LEADER]: Becoming Leader. State: Replica: a8b8bba4de1e4dc8b6efc2878e222ea9, State: Running, Role: LEADER
I20260620 18:41:11.567632 12220 consensus_queue.cc:237] T 00000000000000000000000000000000 P a8b8bba4de1e4dc8b6efc2878e222ea9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8b8bba4de1e4dc8b6efc2878e222ea9" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 33745 } }
I20260620 18:41:11.567955 12210 sys_catalog.cc:564] T 00000000000000000000000000000000 P a8b8bba4de1e4dc8b6efc2878e222ea9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:41:11.570231 12221 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8b8bba4de1e4dc8b6efc2878e222ea9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a8b8bba4de1e4dc8b6efc2878e222ea9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8b8bba4de1e4dc8b6efc2878e222ea9" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 33745 } } }
I20260620 18:41:11.570591 12221 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8b8bba4de1e4dc8b6efc2878e222ea9 [sys.catalog]: This master's current role is: LEADER
I20260620 18:41:11.571065 12222 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8b8bba4de1e4dc8b6efc2878e222ea9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a8b8bba4de1e4dc8b6efc2878e222ea9. Latest consensus state: current_term: 1 leader_uuid: "a8b8bba4de1e4dc8b6efc2878e222ea9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8b8bba4de1e4dc8b6efc2878e222ea9" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 33745 } } }
I20260620 18:41:11.571429 12222 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8b8bba4de1e4dc8b6efc2878e222ea9 [sys.catalog]: This master's current role is: LEADER
I20260620 18:41:11.571660 12227 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:41:11.578022 12227 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:41:11.591816 12227 catalog_manager.cc:1347] Generated new cluster ID: 8febfda9b018479b80fcccbd6d86fd26
I20260620 18:41:11.591934 12227 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:41:11.617089 12227 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:41:11.617568 12227 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:41:11.628319 12227 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a8b8bba4de1e4dc8b6efc2878e222ea9: Generated new TSK 0
I20260620 18:41:11.629308 12227 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:41:11.638309 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 12143
W20260620 18:41:11.639273 12205 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:36037
W20260620 18:41:11.639961 22676 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 12143: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:41:11.640105 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 12143
2026-06-20T18:41:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1869 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980527431848-22676-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:41:11 dist-test-slave-5lgp krb5kdc[12242](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:41:11 dist-test-slave-5lgp krb5kdc[12242](info): set up 2 sockets
Jun 20 18:41:11 dist-test-slave-5lgp krb5kdc[12242](info): commencing operation
krb5kdc: starting...
W20260620 18:41:12.741784 22676 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980527431848-22676-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980527431848-22676-0/krb5kdc/test-user.keytab.
Jun 20 18:41:12 dist-test-slave-5lgp krb5kdc[12242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980872, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:41:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:41:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.37.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.37.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.37.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab.
Entry for principal kudu/127.22.37.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab.
I20260620 18:41:12.993016 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980527431848-22676-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980527431848-22676-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980527431848-22676-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.62:46139
--webserver_interface=127.22.37.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab
--principal=kudu/127.22.37.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:39979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.37.62:46139
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:41:13.318892 12255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:41:13.319187 12255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:41:13.319348 12255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:41:13.326719 12255 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:41:13.326829 12255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:41:13.326973 12255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:41:13.327067 12255 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:41:13.327153 12255 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:41:13.336697 12255 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:39979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980527431848-22676-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980527431848-22676-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.37.62:46139
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab
--principal=kudu/127.22.37.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.62:46139
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.37.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980527431848-22676-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:41:13.338025 12255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:41:13.339365 12255 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:41:13.350970 12260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:13.350999 12261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:13.351033 12263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:41:13.351847 12255 server_base.cc:1034] running on GCE node
I20260620 18:41:13.352612 12255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:41:13.354054 12255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:41:13.355247 12255 hybrid_clock.cc:648] HybridClock initialized: now 1781980873355192 us; error 48 us; skew 500 ppm
I20260620 18:41:13.355572 12255 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:41:13 dist-test-slave-5lgp krb5kdc[12242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980873, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:41:13.358564 12255 init.cc:377] Logged in from keytab as kudu/127.22.37.62@KRBTEST.COM (short username kudu)
I20260620 18:41:13.359977 12255 webserver.cc:460] Webserver started at http://127.22.37.62:35293/ using document root <none> and password file <none>
I20260620 18:41:13.360671 12255 fs_manager.cc:359] Metadata directory not provided
I20260620 18:41:13.360819 12255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:41:13.361127 12255 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:41:13.364105 12255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980527431848-22676-0/minicluster-data/master-0/data/instance:
uuid: "2bec58a194b74c54913e7d1369b172cf"
format_stamp: "Formatted at 2026-06-20 18:41:13 on dist-test-slave-5lgp"
server_key: "38aed484aad6f1c415c9d37da774ec25"
server_key_iv: "c48497bdbe4f48e94ffb74b84c2c5807"
server_key_version: "encryptionkey@0"
I20260620 18:41:13.364853 12255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980527431848-22676-0/minicluster-data/master-0/wal/instance:
uuid: "2bec58a194b74c54913e7d1369b172cf"
format_stamp: "Formatted at 2026-06-20 18:41:13 on dist-test-slave-5lgp"
server_key: "38aed484aad6f1c415c9d37da774ec25"
server_key_iv: "c48497bdbe4f48e94ffb74b84c2c5807"
server_key_version: "encryptionkey@0"
I20260620 18:41:13.369830 12255 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 18:41:13.373055 12270 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:13.374361 12255 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:41:13.374549 12255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980527431848-22676-0/minicluster-data/master-0/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980527431848-22676-0/minicluster-data/master-0/wal
uuid: "2bec58a194b74c54913e7d1369b172cf"
format_stamp: "Formatted at 2026-06-20 18:41:13 on dist-test-slave-5lgp"
server_key: "38aed484aad6f1c415c9d37da774ec25"
server_key_iv: "c48497bdbe4f48e94ffb74b84c2c5807"
server_key_version: "encryptionkey@0"
I20260620 18:41:13.374738 12255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980527431848-22676-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980527431848-22676-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980527431848-22676-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:41:13.393903 12255 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:41:13.398294 12255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:41:13.398584 12255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:41:13.417009 12255 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.62:46139
I20260620 18:41:13.417032 12321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.62:46139 every 8 connection(s)
I20260620 18:41:13.418416 12255 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
I20260620 18:41:13.422123 12322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:41:13.427668 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 12255
I20260620 18:41:13.427845 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980527431848-22676-0/minicluster-data/master-0/wal/instance
I20260620 18:41:13.428376 22676 external_mini_cluster.cc:1428] Setting key 1284feae80fcdbee3fe3f9578d5ec60f
I20260620 18:41:13.433261 12322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bec58a194b74c54913e7d1369b172cf: Bootstrap starting.
I20260620 18:41:13.436949 12322 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2bec58a194b74c54913e7d1369b172cf: Neither blocks nor log segments found. Creating new log.
I20260620 18:41:13.438218 12322 log.cc:824] T 00000000000000000000000000000000 P 2bec58a194b74c54913e7d1369b172cf: Log is configured to *not* fsync() on all Append() calls
I20260620 18:41:13.441704 12322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bec58a194b74c54913e7d1369b172cf: No bootstrap required, opened a new log
Jun 20 18:41:13 dist-test-slave-5lgp krb5kdc[12242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980872, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM
I20260620 18:41:13.449693 12322 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2bec58a194b74c54913e7d1369b172cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bec58a194b74c54913e7d1369b172cf" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 46139 } }
I20260620 18:41:13.450084 12322 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2bec58a194b74c54913e7d1369b172cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:41:13.450206 12322 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2bec58a194b74c54913e7d1369b172cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bec58a194b74c54913e7d1369b172cf, State: Initialized, Role: FOLLOWER
I20260620 18:41:13.450950 12322 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2bec58a194b74c54913e7d1369b172cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bec58a194b74c54913e7d1369b172cf" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 46139 } }
I20260620 18:41:13.451227 12322 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2bec58a194b74c54913e7d1369b172cf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:41:13.451363 12322 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2bec58a194b74c54913e7d1369b172cf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:41:13.451537 12322 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2bec58a194b74c54913e7d1369b172cf [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:41:13.452530 12325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:13.430247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34161 (local address 127.22.37.62:46139)
0620 18:41:13.431102 (+ 855us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:13.431117 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:13.431164 (+ 47us) server_negotiation.cc:407] Connection header received
0620 18:41:13.432808 (+ 1644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:13.432830 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:13.433226 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:13.433632 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:13.435497 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:13.436758 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:13.438578 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:13.439690 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:13.439816 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:13.444643 (+ 4827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:13.444652 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:13.444668 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:13.444712 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:13.447872 (+ 3160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:13.448574 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:13.448581 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:13.448587 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:13.448764 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:13.449174 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:13.449178 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:13.449181 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:13.449761 (+ 580us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:13.450252 (+ 491us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:13.450671 (+ 419us) server_negotiation.cc:299] Negotiation successful
0620 18:41:13.451103 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":412,"threads_started":1}
I20260620 18:41:13.454599 12322 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2bec58a194b74c54913e7d1369b172cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bec58a194b74c54913e7d1369b172cf" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 46139 } }
I20260620 18:41:13.455148 12322 leader_election.cc:302] T 00000000000000000000000000000000 P 2bec58a194b74c54913e7d1369b172cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2bec58a194b74c54913e7d1369b172cf; no voters:
I20260620 18:41:13.455854 12322 leader_election.cc:288] T 00000000000000000000000000000000 P 2bec58a194b74c54913e7d1369b172cf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:41:13.455967 12327 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2bec58a194b74c54913e7d1369b172cf [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:41:13.456346 22676 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:41:13.457906 12322 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2bec58a194b74c54913e7d1369b172cf [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:41:13.462973 12327 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2bec58a194b74c54913e7d1369b172cf [term 1 LEADER]: Becoming Leader. State: Replica: 2bec58a194b74c54913e7d1369b172cf, State: Running, Role: LEADER
I20260620 18:41:13.463579 12327 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2bec58a194b74c54913e7d1369b172cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bec58a194b74c54913e7d1369b172cf" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 46139 } }
I20260620 18:41:13.466503 12328 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bec58a194b74c54913e7d1369b172cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2bec58a194b74c54913e7d1369b172cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bec58a194b74c54913e7d1369b172cf" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 46139 } } }
I20260620 18:41:13.466468 12329 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bec58a194b74c54913e7d1369b172cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2bec58a194b74c54913e7d1369b172cf. Latest consensus state: current_term: 1 leader_uuid: "2bec58a194b74c54913e7d1369b172cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bec58a194b74c54913e7d1369b172cf" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 46139 } } }
I20260620 18:41:13.466893 12329 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bec58a194b74c54913e7d1369b172cf [sys.catalog]: This master's current role is: LEADER
I20260620 18:41:13.466893 12328 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bec58a194b74c54913e7d1369b172cf [sys.catalog]: This master's current role is: LEADER
I20260620 18:41:13.482892 12339 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:41:13.485117 12325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:13.463364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47707 (local address 127.22.37.62:46139)
0620 18:41:13.463559 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:13.463568 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:13.464503 (+ 935us) server_negotiation.cc:407] Connection header received
0620 18:41:13.464681 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:13.464690 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:13.464790 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:13.464910 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:13.467320 (+ 2410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:13.468825 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:13.471235 (+ 2410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:13.472213 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:13.472364 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:13.476375 (+ 4011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:13.476385 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:13.476393 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:13.476441 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:13.481105 (+ 4664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:13.482180 (+ 1075us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:13.482187 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:13.482191 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:13.482414 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:13.482892 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:13.482899 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:13.482902 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:13.483368 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:13.483605 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:13.484277 (+ 672us) server_negotiation.cc:299] Negotiation successful
0620 18:41:13.484590 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 18:41:13.488260 12339 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:41:13.498462 12339 catalog_manager.cc:1347] Generated new cluster ID: 40ccdc22d0b94f128af53cf2298fff6f
I20260620 18:41:13.498580 12339 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:41:13.518544 12339 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:41:13.520458 12339 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:41:13.528374 12339 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2bec58a194b74c54913e7d1369b172cf: Generated new TSK 0
I20260620 18:41:13.529528 12339 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:41:13.538918 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 12255
I20260620 18:41:13.539911 12317 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:41:13.629241 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 12255
2026-06-20T18:41:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1987 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980527431848-22676-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:41:13 dist-test-slave-5lgp krb5kdc[12354](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:41:13 dist-test-slave-5lgp krb5kdc[12354](info): set up 2 sockets
Jun 20 18:41:13 dist-test-slave-5lgp krb5kdc[12354](info): commencing operation
krb5kdc: starting...
W20260620 18:41:14.726590 22676 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980527431848-22676-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980527431848-22676-0/krb5kdc/test-user.keytab.
Jun 20 18:41:14 dist-test-slave-5lgp krb5kdc[12354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:41:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:41:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.37.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.37.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.37.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab.
Entry for principal kudu/127.22.37.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab.
I20260620 18:41:14.977056 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980527431848-22676-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980527431848-22676-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980527431848-22676-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.62:45877
--webserver_interface=127.22.37.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab
--principal=kudu/127.22.37.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:44861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.37.62:45877
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:41:15.304507 12367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:41:15.304858 12367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:41:15.305019 12367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:41:15.312423 12367 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:41:15.312580 12367 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:41:15.312682 12367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:41:15.312788 12367 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:41:15.312862 12367 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:41:15.322656 12367 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:44861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980527431848-22676-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980527431848-22676-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.37.62:45877
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab
--principal=kudu/127.22.37.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.62:45877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.37.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980527431848-22676-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:41:15.324052 12367 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:41:15.325415 12367 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:41:15.337006 12373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:15.337109 12372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:15.337018 12375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:41:15.337836 12367 server_base.cc:1034] running on GCE node
I20260620 18:41:15.338611 12367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:41:15.340044 12367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:41:15.341241 12367 hybrid_clock.cc:648] HybridClock initialized: now 1781980875341191 us; error 38 us; skew 500 ppm
I20260620 18:41:15.341603 12367 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:41:15 dist-test-slave-5lgp krb5kdc[12354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980875, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:41:15.348744 12367 init.cc:377] Logged in from keytab as kudu/127.22.37.62@KRBTEST.COM (short username kudu)
I20260620 18:41:15.350226 12367 webserver.cc:460] Webserver started at http://127.22.37.62:34625/ using document root <none> and password file <none>
I20260620 18:41:15.350891 12367 fs_manager.cc:359] Metadata directory not provided
I20260620 18:41:15.351044 12367 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:41:15.351370 12367 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:41:15.354426 12367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980527431848-22676-0/minicluster-data/master-0/data/instance:
uuid: "a7d0711402174c44b1e1be8106ac75fb"
format_stamp: "Formatted at 2026-06-20 18:41:15 on dist-test-slave-5lgp"
server_key: "0c4c88a98a869b16d6da97e1580aae7d"
server_key_iv: "cbf4fedf3afd585015eb593d15c23d40"
server_key_version: "encryptionkey@0"
I20260620 18:41:15.355206 12367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980527431848-22676-0/minicluster-data/master-0/wal/instance:
uuid: "a7d0711402174c44b1e1be8106ac75fb"
format_stamp: "Formatted at 2026-06-20 18:41:15 on dist-test-slave-5lgp"
server_key: "0c4c88a98a869b16d6da97e1580aae7d"
server_key_iv: "cbf4fedf3afd585015eb593d15c23d40"
server_key_version: "encryptionkey@0"
I20260620 18:41:15.360363 12367 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 18:41:15.363917 12382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:15.365272 12367 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:41:15.365475 12367 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980527431848-22676-0/minicluster-data/master-0/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980527431848-22676-0/minicluster-data/master-0/wal
uuid: "a7d0711402174c44b1e1be8106ac75fb"
format_stamp: "Formatted at 2026-06-20 18:41:15 on dist-test-slave-5lgp"
server_key: "0c4c88a98a869b16d6da97e1580aae7d"
server_key_iv: "cbf4fedf3afd585015eb593d15c23d40"
server_key_version: "encryptionkey@0"
I20260620 18:41:15.365684 12367 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980527431848-22676-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980527431848-22676-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980527431848-22676-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:41:15.412001 12367 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:41:15.416141 12367 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:41:15.416441 12367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:41:15.435316 12367 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.62:45877
I20260620 18:41:15.435338 12433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.62:45877 every 8 connection(s)
I20260620 18:41:15.436697 12367 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
I20260620 18:41:15.440358 12434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:41:15.440784 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 12367
I20260620 18:41:15.440979 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980527431848-22676-0/minicluster-data/master-0/wal/instance
I20260620 18:41:15.441506 22676 external_mini_cluster.cc:1428] Setting key 2666a283a0acb13cfcf0bdcb72208457
I20260620 18:41:15.452404 12434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7d0711402174c44b1e1be8106ac75fb: Bootstrap starting.
I20260620 18:41:15.454634 12437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:15.443319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58035 (local address 127.22.37.62:45877)
0620 18:41:15.444087 (+ 768us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:15.444098 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:15.444140 (+ 42us) server_negotiation.cc:407] Connection header received
0620 18:41:15.446423 (+ 2283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:15.446447 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:15.446835 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:15.447246 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:15.447756 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:15.448980 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:15.450387 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:15.451049 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:15.451257 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:15.452451 (+ 1194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:15.452460 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:15.452474 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:41:15.453007 (+ 533us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:15.453078 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:41:15.453100 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:15.453227 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:15.453457 (+ 230us) server_negotiation.cc:299] Negotiation successful
0620 18:41:15.453703 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":328,"threads_started":1}
I20260620 18:41:15.455765 12434 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a7d0711402174c44b1e1be8106ac75fb: Neither blocks nor log segments found. Creating new log.
I20260620 18:41:15.456887 12434 log.cc:824] T 00000000000000000000000000000000 P a7d0711402174c44b1e1be8106ac75fb: Log is configured to *not* fsync() on all Append() calls
I20260620 18:41:15.458272 22676 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:41:15.459856 12434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7d0711402174c44b1e1be8106ac75fb: No bootstrap required, opened a new log
I20260620 18:41:15.470084 12437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:15.463448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46543 (local address 127.22.37.62:45877)
0620 18:41:15.463620 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:15.463625 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:15.464165 (+ 540us) server_negotiation.cc:407] Connection header received
0620 18:41:15.464311 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:15.464316 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:15.464393 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:15.464511 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:15.464972 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:15.466118 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:15.467369 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:15.468091 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:15.468253 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:15.469217 (+ 964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:15.469221 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:15.469225 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:41:15.469240 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:15.469287 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:41:15.469300 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:15.469425 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:15.469586 (+ 161us) server_negotiation.cc:299] Negotiation successful
0620 18:41:15.469667 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 18:41:15.469853 12434 raft_consensus.cc:348] T 00000000000000000000000000000000 P a7d0711402174c44b1e1be8106ac75fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7d0711402174c44b1e1be8106ac75fb" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 45877 } }
I20260620 18:41:15.470292 12434 raft_consensus.cc:374] T 00000000000000000000000000000000 P a7d0711402174c44b1e1be8106ac75fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:41:15.470439 12434 raft_consensus.cc:729] T 00000000000000000000000000000000 P a7d0711402174c44b1e1be8106ac75fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7d0711402174c44b1e1be8106ac75fb, State: Initialized, Role: FOLLOWER
I20260620 18:41:15.471092 12434 consensus_queue.cc:260] T 00000000000000000000000000000000 P a7d0711402174c44b1e1be8106ac75fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7d0711402174c44b1e1be8106ac75fb" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 45877 } }
I20260620 18:41:15.471397 12434 raft_consensus.cc:388] T 00000000000000000000000000000000 P a7d0711402174c44b1e1be8106ac75fb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:41:15.471544 12434 raft_consensus.cc:482] T 00000000000000000000000000000000 P a7d0711402174c44b1e1be8106ac75fb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:41:15.471696 12434 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a7d0711402174c44b1e1be8106ac75fb [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:41:15.473770 12434 raft_consensus.cc:504] T 00000000000000000000000000000000 P a7d0711402174c44b1e1be8106ac75fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7d0711402174c44b1e1be8106ac75fb" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 45877 } }
I20260620 18:41:15.474181 12434 leader_election.cc:302] T 00000000000000000000000000000000 P a7d0711402174c44b1e1be8106ac75fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a7d0711402174c44b1e1be8106ac75fb; no voters:
I20260620 18:41:15.474637 12434 leader_election.cc:288] T 00000000000000000000000000000000 P a7d0711402174c44b1e1be8106ac75fb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:41:15.474772 12444 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a7d0711402174c44b1e1be8106ac75fb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:41:15.475451 12444 raft_consensus.cc:686] T 00000000000000000000000000000000 P a7d0711402174c44b1e1be8106ac75fb [term 1 LEADER]: Becoming Leader. State: Replica: a7d0711402174c44b1e1be8106ac75fb, State: Running, Role: LEADER
I20260620 18:41:15.475952 12444 consensus_queue.cc:237] T 00000000000000000000000000000000 P a7d0711402174c44b1e1be8106ac75fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7d0711402174c44b1e1be8106ac75fb" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 45877 } }
I20260620 18:41:15.476593 12434 sys_catalog.cc:564] T 00000000000000000000000000000000 P a7d0711402174c44b1e1be8106ac75fb [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:41:15.478543 12446 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7d0711402174c44b1e1be8106ac75fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7d0711402174c44b1e1be8106ac75fb. Latest consensus state: current_term: 1 leader_uuid: "a7d0711402174c44b1e1be8106ac75fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7d0711402174c44b1e1be8106ac75fb" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 45877 } } }
I20260620 18:41:15.478837 12446 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7d0711402174c44b1e1be8106ac75fb [sys.catalog]: This master's current role is: LEADER
I20260620 18:41:15.480329 12450 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:41:15.480401 12445 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7d0711402174c44b1e1be8106ac75fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a7d0711402174c44b1e1be8106ac75fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7d0711402174c44b1e1be8106ac75fb" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 45877 } } }
I20260620 18:41:15.480702 12445 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7d0711402174c44b1e1be8106ac75fb [sys.catalog]: This master's current role is: LEADER
I20260620 18:41:15.486680 12450 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:41:15.494390 12450 catalog_manager.cc:1347] Generated new cluster ID: 86b1ff6555514ad59b145f289370e8cc
I20260620 18:41:15.494511 12450 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:41:15.510316 12450 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:41:15.512147 12450 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:41:15.524633 12450 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a7d0711402174c44b1e1be8106ac75fb: Generated new TSK 0
I20260620 18:41:15.525807 12450 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:41:15.539223 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 12367
W20260620 18:41:15.540611 12429 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:58035
W20260620 18:41:15.541343 22676 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 12367: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:41:15.541532 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 12367
2026-06-20T18:41:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1911 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980527431848-22676-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:41:15 dist-test-slave-5lgp krb5kdc[12465](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:41:15 dist-test-slave-5lgp krb5kdc[12465](info): set up 2 sockets
Jun 20 18:41:15 dist-test-slave-5lgp krb5kdc[12465](info): commencing operation
krb5kdc: starting...
W20260620 18:41:16.641964 22676 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980527431848-22676-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980527431848-22676-0/krb5kdc/test-user.keytab.
Jun 20 18:41:16 dist-test-slave-5lgp krb5kdc[12465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980876, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:41:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:41:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.37.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.37.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.37.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab.
Entry for principal kudu/127.22.37.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab.
I20260620 18:41:16.900520 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980527431848-22676-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980527431848-22676-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980527431848-22676-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.62:37919
--webserver_interface=127.22.37.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab
--principal=kudu/127.22.37.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:33189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.37.62:37919
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:41:17.235502 12478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:41:17.235788 12478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:41:17.235944 12478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:41:17.242935 12478 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:41:17.243042 12478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:41:17.243176 12478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:41:17.243288 12478 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:41:17.243350 12478 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:41:17.252940 12478 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:33189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980527431848-22676-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980527431848-22676-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.37.62:37919
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab
--principal=kudu/127.22.37.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.62:37919
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.37.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980527431848-22676-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:41:17.254284 12478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:41:17.255609 12478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:41:17.267230 12483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:17.267262 12484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:17.267499 12486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:41:17.268487 12478 server_base.cc:1034] running on GCE node
I20260620 18:41:17.269322 12478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:41:17.270923 12478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:41:17.272156 12478 hybrid_clock.cc:648] HybridClock initialized: now 1781980877272110 us; error 39 us; skew 500 ppm
I20260620 18:41:17.272567 12478 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:41:17 dist-test-slave-5lgp krb5kdc[12465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980877, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:41:17.276021 12478 init.cc:377] Logged in from keytab as kudu/127.22.37.62@KRBTEST.COM (short username kudu)
I20260620 18:41:17.277772 12478 webserver.cc:460] Webserver started at http://127.22.37.62:35507/ using document root <none> and password file <none>
I20260620 18:41:17.278546 12478 fs_manager.cc:359] Metadata directory not provided
I20260620 18:41:17.278707 12478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:41:17.279050 12478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:41:17.282395 12478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980527431848-22676-0/minicluster-data/master-0/data/instance:
uuid: "ae56f58a32be4f78818f41c58f1365eb"
format_stamp: "Formatted at 2026-06-20 18:41:17 on dist-test-slave-5lgp"
server_key: "7813e60e96aecb09d2cec60a7bc03f08"
server_key_iv: "03c463fab56ca38621a7da9503bfdf38"
server_key_version: "encryptionkey@0"
I20260620 18:41:17.283260 12478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980527431848-22676-0/minicluster-data/master-0/wal/instance:
uuid: "ae56f58a32be4f78818f41c58f1365eb"
format_stamp: "Formatted at 2026-06-20 18:41:17 on dist-test-slave-5lgp"
server_key: "7813e60e96aecb09d2cec60a7bc03f08"
server_key_iv: "03c463fab56ca38621a7da9503bfdf38"
server_key_version: "encryptionkey@0"
I20260620 18:41:17.289105 12478 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 18:41:17.292948 12493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:17.294489 12478 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:41:17.294680 12478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980527431848-22676-0/minicluster-data/master-0/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980527431848-22676-0/minicluster-data/master-0/wal
uuid: "ae56f58a32be4f78818f41c58f1365eb"
format_stamp: "Formatted at 2026-06-20 18:41:17 on dist-test-slave-5lgp"
server_key: "7813e60e96aecb09d2cec60a7bc03f08"
server_key_iv: "03c463fab56ca38621a7da9503bfdf38"
server_key_version: "encryptionkey@0"
I20260620 18:41:17.294909 12478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980527431848-22676-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980527431848-22676-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980527431848-22676-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:41:17.300799 12478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:41:17.305272 12478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:41:17.305601 12478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:41:17.323791 12478 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.62:37919
I20260620 18:41:17.323856 12544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.62:37919 every 8 connection(s)
I20260620 18:41:17.325228 12478 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
I20260620 18:41:17.327553 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 12478
I20260620 18:41:17.327737 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980527431848-22676-0/minicluster-data/master-0/wal/instance
I20260620 18:41:17.328213 22676 external_mini_cluster.cc:1428] Setting key 5239cc24bc84e123f8e4ec2051ea1522
I20260620 18:41:17.329511 12545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:41:17.337311 12548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:17.330063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38321 (local address 127.22.37.62:37919)
0620 18:41:17.330874 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:17.330889 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:17.330945 (+ 56us) server_negotiation.cc:407] Connection header received
0620 18:41:17.333009 (+ 2064us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:17.333037 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:17.333537 (+ 500us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:17.334023 (+ 486us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:17.335249 (+ 1226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:17.335257 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:17.335274 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:41:17.335293 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:17.335393 (+ 100us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:41:17.335420 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:17.335642 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:17.335776 (+ 134us) server_negotiation.cc:299] Negotiation successful
0620 18:41:17.335989 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":324,"threads_started":1}
I20260620 18:41:17.340996 22676 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:41:17.344637 12545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae56f58a32be4f78818f41c58f1365eb: Bootstrap starting.
I20260620 18:41:17.348634 12545 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae56f58a32be4f78818f41c58f1365eb: Neither blocks nor log segments found. Creating new log.
I20260620 18:41:17.348788 12548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:17.346334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53021 (local address 127.22.37.62:37919)
0620 18:41:17.346547 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:17.346555 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:17.346863 (+ 308us) server_negotiation.cc:407] Connection header received
0620 18:41:17.346988 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:17.346994 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:17.347067 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:17.347170 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:17.348076 (+ 906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:17.348081 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:17.348086 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:41:17.348102 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:17.348149 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:41:17.348170 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:17.348282 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:17.348386 (+ 104us) server_negotiation.cc:299] Negotiation successful
0620 18:41:17.348447 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 18:41:17.350032 12545 log.cc:824] T 00000000000000000000000000000000 P ae56f58a32be4f78818f41c58f1365eb: Log is configured to *not* fsync() on all Append() calls
I20260620 18:41:17.352898 12545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae56f58a32be4f78818f41c58f1365eb: No bootstrap required, opened a new log
I20260620 18:41:17.361742 12545 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae56f58a32be4f78818f41c58f1365eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae56f58a32be4f78818f41c58f1365eb" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 37919 } }
I20260620 18:41:17.362134 12545 raft_consensus.cc:374] T 00000000000000000000000000000000 P ae56f58a32be4f78818f41c58f1365eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:41:17.362234 12545 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae56f58a32be4f78818f41c58f1365eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae56f58a32be4f78818f41c58f1365eb, State: Initialized, Role: FOLLOWER
I20260620 18:41:17.362818 12545 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae56f58a32be4f78818f41c58f1365eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae56f58a32be4f78818f41c58f1365eb" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 37919 } }
I20260620 18:41:17.363042 12545 raft_consensus.cc:388] T 00000000000000000000000000000000 P ae56f58a32be4f78818f41c58f1365eb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:41:17.363173 12545 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae56f58a32be4f78818f41c58f1365eb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:41:17.363340 12545 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae56f58a32be4f78818f41c58f1365eb [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:41:17.365571 12545 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae56f58a32be4f78818f41c58f1365eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae56f58a32be4f78818f41c58f1365eb" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 37919 } }
I20260620 18:41:17.366102 12545 leader_election.cc:302] T 00000000000000000000000000000000 P ae56f58a32be4f78818f41c58f1365eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae56f58a32be4f78818f41c58f1365eb; no voters:
I20260620 18:41:17.366688 12545 leader_election.cc:288] T 00000000000000000000000000000000 P ae56f58a32be4f78818f41c58f1365eb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:41:17.366837 12555 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae56f58a32be4f78818f41c58f1365eb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:41:17.367542 12555 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae56f58a32be4f78818f41c58f1365eb [term 1 LEADER]: Becoming Leader. State: Replica: ae56f58a32be4f78818f41c58f1365eb, State: Running, Role: LEADER
I20260620 18:41:17.368100 12555 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae56f58a32be4f78818f41c58f1365eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae56f58a32be4f78818f41c58f1365eb" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 37919 } }
I20260620 18:41:17.368842 12545 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae56f58a32be4f78818f41c58f1365eb [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:41:17.371362 12556 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae56f58a32be4f78818f41c58f1365eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ae56f58a32be4f78818f41c58f1365eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae56f58a32be4f78818f41c58f1365eb" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 37919 } } }
I20260620 18:41:17.371832 12556 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae56f58a32be4f78818f41c58f1365eb [sys.catalog]: This master's current role is: LEADER
I20260620 18:41:17.373579 12557 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae56f58a32be4f78818f41c58f1365eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae56f58a32be4f78818f41c58f1365eb. Latest consensus state: current_term: 1 leader_uuid: "ae56f58a32be4f78818f41c58f1365eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae56f58a32be4f78818f41c58f1365eb" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 37919 } } }
I20260620 18:41:17.373836 12557 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae56f58a32be4f78818f41c58f1365eb [sys.catalog]: This master's current role is: LEADER
I20260620 18:41:17.374336 12562 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:41:17.379283 12562 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:41:17.388361 12562 catalog_manager.cc:1347] Generated new cluster ID: 3d51bd637b594111879d28e7cb0234b0
I20260620 18:41:17.388506 12562 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:41:17.419417 12562 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:41:17.419932 12562 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:41:17.428428 12562 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ae56f58a32be4f78818f41c58f1365eb: Generated new TSK 0
I20260620 18:41:17.429446 12562 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:41:17.440586 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 12478
W20260620 18:41:17.441668 12540 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:38321
W20260620 18:41:17.442337 22676 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 12478: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:41:17.442492 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 12478
2026-06-20T18:41:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1900 ms)
[----------] 18 tests from AuthTokenIssuingTest (34802 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980527431848-22676-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:41:17 dist-test-slave-5lgp krb5kdc[12576](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:41:17 dist-test-slave-5lgp krb5kdc[12576](info): set up 2 sockets
Jun 20 18:41:17 dist-test-slave-5lgp krb5kdc[12576](info): commencing operation
krb5kdc: starting...
W20260620 18:41:18.554006 22676 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.003s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980527431848-22676-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980527431848-22676-0/krb5kdc/test-user.keytab.
Jun 20 18:41:18 dist-test-slave-5lgp krb5kdc[12576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980878, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:41:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:41:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.37.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.37.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.37.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab.
Entry for principal kudu/127.22.37.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab.
I20260620 18:41:18.804337 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980527431848-22676-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980527431848-22676-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980527431848-22676-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.62:38759
--webserver_interface=127.22.37.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab
--principal=kudu/127.22.37.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:33177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.37.62:38759,127.22.37.61:45785,127.22.37.60:45055
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:41:19.125682 12589 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 18:41:19.126089 12589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:41:19.126215 12589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:41:19.126377 12589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:41:19.133667 12589 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:41:19.133790 12589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:41:19.133880 12589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:41:19.133950 12589 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:41:19.134052 12589 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:41:19.146898 12589 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:33177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980527431848-22676-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980527431848-22676-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.37.62:38759,127.22.37.61:45785,127.22.37.60:45055
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab
--principal=kudu/127.22.37.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.62:38759
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.37.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980527431848-22676-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:41:19.148267 12589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:41:19.149700 12589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:41:19.161612 12594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:19.161679 12595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:19.161700 12597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:41:19.162787 12589 server_base.cc:1034] running on GCE node
I20260620 18:41:19.163547 12589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:41:19.164886 12589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:41:19.166098 12589 hybrid_clock.cc:648] HybridClock initialized: now 1781980879166045 us; error 24 us; skew 500 ppm
I20260620 18:41:19.166442 12589 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:41:19 dist-test-slave-5lgp krb5kdc[12576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980879, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:41:19.169631 12589 init.cc:377] Logged in from keytab as kudu/127.22.37.62@KRBTEST.COM (short username kudu)
I20260620 18:41:19.171046 12589 webserver.cc:460] Webserver started at http://127.22.37.62:41557/ using document root <none> and password file <none>
I20260620 18:41:19.171723 12589 fs_manager.cc:359] Metadata directory not provided
I20260620 18:41:19.171865 12589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:41:19.172185 12589 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:41:19.175203 12589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980527431848-22676-0/minicluster-data/master-0/data/instance:
uuid: "45e9b431d78d454da7c9418b68be52c9"
format_stamp: "Formatted at 2026-06-20 18:41:19 on dist-test-slave-5lgp"
server_key: "a685283d255d92151db3b3aa9e303d4c"
server_key_iv: "4bfce93c535e822254591d58ddf5f581"
server_key_version: "encryptionkey@0"
I20260620 18:41:19.176007 12589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980527431848-22676-0/minicluster-data/master-0/wal/instance:
uuid: "45e9b431d78d454da7c9418b68be52c9"
format_stamp: "Formatted at 2026-06-20 18:41:19 on dist-test-slave-5lgp"
server_key: "a685283d255d92151db3b3aa9e303d4c"
server_key_iv: "4bfce93c535e822254591d58ddf5f581"
server_key_version: "encryptionkey@0"
I20260620 18:41:19.181053 12589 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260620 18:41:19.184489 12604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:19.185906 12589 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:41:19.186106 12589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980527431848-22676-0/minicluster-data/master-0/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980527431848-22676-0/minicluster-data/master-0/wal
uuid: "45e9b431d78d454da7c9418b68be52c9"
format_stamp: "Formatted at 2026-06-20 18:41:19 on dist-test-slave-5lgp"
server_key: "a685283d255d92151db3b3aa9e303d4c"
server_key_iv: "4bfce93c535e822254591d58ddf5f581"
server_key_version: "encryptionkey@0"
I20260620 18:41:19.186323 12589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980527431848-22676-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980527431848-22676-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980527431848-22676-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:41:19.219694 12589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:41:19.224346 12589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:41:19.224640 12589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:41:19.243667 12589 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.62:38759
I20260620 18:41:19.243690 12655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.62:38759 every 8 connection(s)
I20260620 18:41:19.245079 12589 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
I20260620 18:41:19.248155 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 12589
I20260620 18:41:19.248389 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980527431848-22676-0/minicluster-data/master-0/wal/instance
I20260620 18:41:19.249228 22676 external_mini_cluster.cc:1428] Setting key 8caf02170f77b83f37999980b41a1766
I20260620 18:41:19.249446 12656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:41:19.259860 12656 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.37.62" port: 38759 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:41:19 dist-test-slave-5lgp krb5kdc[12576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980879, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.62@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.37.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.37.61@KRBTEST.COM" created.
I20260620 18:41:19.282028 12659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:19.263245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:38759 (local address 127.0.0.1:36106)
0620 18:41:19.264133 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:19.264179 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:19.266306 (+ 2127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:19.267153 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:19.267160 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:19.267846 (+ 686us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:41:19.268173 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:19.268190 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:19.269764 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:19.269775 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:19.270826 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:19.270839 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:19.271727 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:19.271734 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:19.271901 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:19.272611 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:41:19.272632 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:41:19.274771 (+ 2139us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:41:19.277980 (+ 3209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:19.277998 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:19.278006 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:19.278501 (+ 495us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:19.279008 (+ 507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:19.279011 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:19.279013 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:19.279172 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:19.280156 (+ 984us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:41:19.280162 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:41:19.280347 (+ 185us) client_negotiation.cc:769] Sending connection context
0620 18:41:19.280561 (+ 214us) client_negotiation.cc:240] Negotiation successful
0620 18:41:19.280945 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":406,"threads_started":1}
I20260620 18:41:19.282081 12660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:19.263246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36106 (local address 127.22.37.62:38759)
0620 18:41:19.265013 (+ 1767us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:19.265021 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:19.265071 (+ 50us) server_negotiation.cc:407] Connection header received
0620 18:41:19.266704 (+ 1633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:19.266709 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:19.266933 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:19.267136 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:19.268386 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:19.269619 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:19.270965 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:19.271509 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:19.271705 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:19.275006 (+ 3301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:19.275013 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:19.275024 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:19.275050 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:19.277819 (+ 2769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:19.278633 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:19.278644 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:19.278648 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:19.278886 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:19.279293 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:19.279298 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:19.279303 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:19.280008 (+ 705us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:19.280206 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:19.280757 (+ 551us) server_negotiation.cc:299] Negotiation successful
0620 18:41:19.281145 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":71,"server-negotiator.queue_time_us":1514,"thread_start_us":1275,"threads_started":1}
I20260620 18:41:19.284843 12656 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.37.61" port: 45785 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:41:19.285972 12659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:19.285560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.61:45785 (local address 127.0.0.1:40080)
0620 18:41:19.285756 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:19.285851 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.61:45785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260620 18:41:19.286244 12607 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.37.61:45785: connect: Connection refused (error 111)
I20260620 18:41:19.287863 12659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:19.287480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.61:45785 (local address 127.0.0.1:40092)
0620 18:41:19.287682 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:19.287748 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.61:45785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 18:41:19.288107 12656 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.37.61:45785: Network error: Client connection negotiation failed: client connection to 127.22.37.61:45785: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.37.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.61.keytab.
Entry for principal kudu/127.22.37.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.61.keytab.
I20260620 18:41:19.307816 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980527431848-22676-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980527431848-22676-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980527431848-22676-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980527431848-22676-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.61:45785
--webserver_interface=127.22.37.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.61.keytab
--principal=kudu/127.22.37.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:33177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.37.62:38759,127.22.37.61:45785,127.22.37.60:45055
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:41:19.337469 12656 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.37.61" port: 45785 } attempt: 1
I20260620 18:41:19.338702 12659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:19.338215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.61:45785 (local address 127.0.0.1:40102)
0620 18:41:19.338429 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:19.338537 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.61:45785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260620 18:41:19.339742 12659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:19.339374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.61:45785 (local address 127.0.0.1:40114)
0620 18:41:19.339515 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:19.339595 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.61:45785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 18:41:19.339918 12656 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.37.61:45785: Network error: Client connection negotiation failed: client connection to 127.22.37.61:45785: connect: Connection refused (error 111)
I20260620 18:41:19.408185 12656 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.37.61" port: 45785 } attempt: 2
I20260620 18:41:19.409484 12659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:19.408876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.61:45785 (local address 127.0.0.1:40126)
0620 18:41:19.409139 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:19.409252 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.61:45785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260620 18:41:19.410507 12659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:19.410111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.61:45785 (local address 127.0.0.1:40128)
0620 18:41:19.410258 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:19.410342 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.61:45785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 18:41:19.410730 12656 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.37.61:45785: Network error: Client connection negotiation failed: client connection to 127.22.37.61:45785: connect: Connection refused (error 111)
I20260620 18:41:19.502013 12656 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.37.61" port: 45785 } attempt: 3
I20260620 18:41:19.503372 12659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:19.502879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.61:45785 (local address 127.0.0.1:40134)
0620 18:41:19.503125 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:19.503223 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.61:45785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260620 18:41:19.504662 12659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:19.504224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.61:45785 (local address 127.0.0.1:40150)
0620 18:41:19.504435 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:19.504511 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.61:45785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260620 18:41:19.504894 12656 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.37.61:45785: Network error: Client connection negotiation failed: client connection to 127.22.37.61:45785: connect: Connection refused (error 111)
W20260620 18:41:19.645053 12663 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 18:41:19.645498 12663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:41:19.645632 12663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:41:19.645767 12663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 18:41:19.648147 12656 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.37.61" port: 45785 } attempt: 4
I20260620 18:41:19.649428 12659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:19.648802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.61:45785 (local address 127.0.0.1:40164)
0620 18:41:19.649108 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:19.649247 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.61:45785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173}
I20260620 18:41:19.650549 12659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:19.650165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.61:45785 (local address 127.0.0.1:40180)
0620 18:41:19.650317 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:19.650414 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.61:45785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 18:41:19.650810 12656 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.37.61:45785: Network error: Client connection negotiation failed: client connection to 127.22.37.61:45785: connect: Connection refused (error 111)
W20260620 18:41:19.653923 12663 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:41:19.654009 12663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:41:19.654138 12663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:41:19.654197 12663 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:41:19.654279 12663 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:41:19.664886 12663 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:33177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980527431848-22676-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980527431848-22676-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.22.37.62:38759,127.22.37.61:45785,127.22.37.60:45055
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.61.keytab
--principal=kudu/127.22.37.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.61:45785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980527431848-22676-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.22.37.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980527431848-22676-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:41:19.666368 12663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:41:19.667692 12663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:41:19.679781 12671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:19.679821 12668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:19.680287 12669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:41:19.680562 12663 server_base.cc:1034] running on GCE node
I20260620 18:41:19.681452 12663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:41:19.682858 12663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:41:19.684105 12663 hybrid_clock.cc:648] HybridClock initialized: now 1781980879684025 us; error 56 us; skew 500 ppm
I20260620 18:41:19.684432 12663 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:41:19 dist-test-slave-5lgp krb5kdc[12576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980879, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:41:19.688254 12663 init.cc:377] Logged in from keytab as kudu/127.22.37.61@KRBTEST.COM (short username kudu)
I20260620 18:41:19.690749 12663 webserver.cc:460] Webserver started at http://127.22.37.61:44815/ using document root <none> and password file <none>
I20260620 18:41:19.691767 12663 fs_manager.cc:359] Metadata directory not provided
I20260620 18:41:19.691936 12663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:41:19.692358 12663 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:41:19.696494 12663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980527431848-22676-0/minicluster-data/master-1/data/instance:
uuid: "58601e366a404d81859c358d347cf516"
format_stamp: "Formatted at 2026-06-20 18:41:19 on dist-test-slave-5lgp"
server_key: "0e27ee1251300057fdedf2bb3967460e"
server_key_iv: "41738b8a320c7730737e99ad5d0aee2b"
server_key_version: "encryptionkey@0"
I20260620 18:41:19.697578 12663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980527431848-22676-0/minicluster-data/master-1/wal/instance:
uuid: "58601e366a404d81859c358d347cf516"
format_stamp: "Formatted at 2026-06-20 18:41:19 on dist-test-slave-5lgp"
server_key: "0e27ee1251300057fdedf2bb3967460e"
server_key_iv: "41738b8a320c7730737e99ad5d0aee2b"
server_key_version: "encryptionkey@0"
I20260620 18:41:19.705426 12663 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260620 18:41:19.710628 12678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:19.712615 12663 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 18:41:19.712817 12663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980527431848-22676-0/minicluster-data/master-1/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980527431848-22676-0/minicluster-data/master-1/wal
uuid: "58601e366a404d81859c358d347cf516"
format_stamp: "Formatted at 2026-06-20 18:41:19 on dist-test-slave-5lgp"
server_key: "0e27ee1251300057fdedf2bb3967460e"
server_key_iv: "41738b8a320c7730737e99ad5d0aee2b"
server_key_version: "encryptionkey@0"
I20260620 18:41:19.713078 12663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980527431848-22676-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980527431848-22676-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980527431848-22676-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:41:19.739550 12663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:41:19.744053 12663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:41:19.744288 12663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:41:19.762858 12663 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.61:45785
I20260620 18:41:19.763001 12729 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.61:45785 every 8 connection(s)
I20260620 18:41:19.764183 12663 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980527431848-22676-0/minicluster-data/master-1/data/info.pb
I20260620 18:41:19.767786 12730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:41:19.772991 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 12663
I20260620 18:41:19.773187 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980527431848-22676-0/minicluster-data/master-1/wal/instance
I20260620 18:41:19.773716 22676 external_mini_cluster.cc:1428] Setting key 240dc4387b1a2a7dd7c7d891134d6c24
I20260620 18:41:19.775039 12730 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.37.62" port: 38759 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:41:19 dist-test-slave-5lgp krb5kdc[12576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980879, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.61@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM
I20260620 18:41:19.799736 12660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:19.778264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36122 (local address 127.22.37.62:38759)
0620 18:41:19.780026 (+ 1762us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:19.780034 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:19.780059 (+ 25us) server_negotiation.cc:407] Connection header received
0620 18:41:19.782712 (+ 2653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:19.782719 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:19.782803 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:19.782902 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:19.784688 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:19.785591 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:19.788263 (+ 2672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:19.788758 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:19.788892 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:19.793617 (+ 4725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:19.793622 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:19.793625 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:19.793660 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:19.796278 (+ 2618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:19.797229 (+ 951us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:19.797233 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:19.797235 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:19.797349 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:19.797929 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:19.797939 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:19.797941 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:19.798224 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:19.798347 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:19.799331 (+ 984us) server_negotiation.cc:299] Negotiation successful
0620 18:41:19.799502 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1608}
I20260620 18:41:19.801450 12732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:19.778762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:38759 (local address 127.0.0.1:36122)
0620 18:41:19.779653 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:19.779701 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:19.782431 (+ 2730us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:19.783159 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:19.783173 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:19.783949 (+ 776us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:41:19.784501 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:19.784518 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:19.786004 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:19.786012 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:19.788096 (+ 2084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:19.788112 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:19.789000 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:19.789009 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:19.789331 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:19.790154 (+ 823us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:41:19.790192 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:41:19.793325 (+ 3133us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:41:19.796429 (+ 3104us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:19.796442 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:19.796463 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:19.797115 (+ 652us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:19.797508 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:19.797517 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:19.797523 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:19.797803 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:19.798366 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:41:19.798377 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:41:19.798904 (+ 527us) client_negotiation.cc:769] Sending connection context
0620 18:41:19.799264 (+ 360us) client_negotiation.cc:240] Negotiation successful
0620 18:41:19.799900 (+ 636us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":363,"threads_started":1}
WARNING: no policy specified for kudu/127.22.37.60@KRBTEST.COM; defaulting to no policy
I20260620 18:41:19.803356 12730 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.37.61" port: 45785 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.37.60@KRBTEST.COM" created.
Jun 20 18:41:19 dist-test-slave-5lgp krb5kdc[12576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980879, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.61@KRBTEST.COM for kudu/127.22.37.61@KRBTEST.COM
I20260620 18:41:19.819798 12734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:19.804354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40190 (local address 127.22.37.61:45785)
0620 18:41:19.805015 (+ 661us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:19.805023 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:19.805054 (+ 31us) server_negotiation.cc:407] Connection header received
0620 18:41:19.805113 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:19.805123 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:19.805429 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:19.805548 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:19.806512 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:19.807619 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:19.809434 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:19.809937 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:19.810040 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:19.813815 (+ 3775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:19.813830 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:19.813851 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:19.813898 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:19.816774 (+ 2876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:19.817592 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:19.817598 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:19.817600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:19.817704 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:19.818157 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:19.818160 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:19.818162 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:19.818545 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:19.818707 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:19.819170 (+ 463us) server_negotiation.cc:299] Negotiation successful
0620 18:41:19.819495 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":321,"threads_started":1}
I20260620 18:41:19.819808 12732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:19.804086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.61:45785 (local address 127.0.0.1:40190)
0620 18:41:19.804268 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:19.804284 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:19.804411 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:19.805592 (+ 1181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:19.805599 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:19.806122 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:41:19.806332 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:19.806346 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:19.807765 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:19.807774 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:19.809223 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:19.809239 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:19.810099 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:19.810106 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:19.810317 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:19.810885 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:41:19.810911 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:41:19.813600 (+ 2689us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:41:19.816901 (+ 3301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:19.816910 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:19.816916 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:19.817476 (+ 560us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:19.817825 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:19.817832 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:19.817838 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:19.818017 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:19.818675 (+ 658us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:41:19.818683 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:41:19.818846 (+ 163us) client_negotiation.cc:769] Sending connection context
0620 18:41:19.818991 (+ 145us) client_negotiation.cc:240] Negotiation successful
0620 18:41:19.819338 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 18:41:19.823318 12730 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.37.60" port: 45055 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:41:19.824551 12732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:19.823977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.60:45055 (local address 127.0.0.1:46014)
0620 18:41:19.824186 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:19.824350 (+ 164us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.60:45055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260620 18:41:19.824824 12681 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.37.60:45055: connect: Connection refused (error 111)
I20260620 18:41:19.826541 12732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:19.826093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.60:45055 (local address 127.0.0.1:46016)
0620 18:41:19.826274 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:19.826389 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.60:45055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 18:41:19.826771 12730 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.37.60:45055: Network error: Client connection negotiation failed: client connection to 127.22.37.60:45055: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.37.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.60.keytab.
Entry for principal kudu/127.22.37.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.60.keytab.
I20260620 18:41:19.840464 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980527431848-22676-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980527431848-22676-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980527431848-22676-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980527431848-22676-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.60:45055
--webserver_interface=127.22.37.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.60.keytab
--principal=kudu/127.22.37.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:33177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.37.62:38759,127.22.37.61:45785,127.22.37.60:45055
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:41:19.876102 12730 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.37.60" port: 45055 } attempt: 1
I20260620 18:41:19.877233 12732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:19.876779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.60:45055 (local address 127.0.0.1:46028)
0620 18:41:19.876984 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:19.877087 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.60:45055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260620 18:41:19.878350 12732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:19.878007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.60:45055 (local address 127.0.0.1:46032)
0620 18:41:19.878176 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:19.878247 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.60:45055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260620 18:41:19.878554 12730 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.37.60:45055: Network error: Client connection negotiation failed: client connection to 127.22.37.60:45055: connect: Connection refused (error 111)
I20260620 18:41:19.946832 12730 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.37.60" port: 45055 } attempt: 2
I20260620 18:41:19.948279 12732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:19.947606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.60:45055 (local address 127.0.0.1:46034)
0620 18:41:19.947899 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:19.948049 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.60:45055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260620 18:41:19.949349 12732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:19.948957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.60:45055 (local address 127.0.0.1:46040)
0620 18:41:19.949128 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:19.949211 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.60:45055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260620 18:41:19.949617 12730 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.37.60:45055: Network error: Client connection negotiation failed: client connection to 127.22.37.60:45055: connect: Connection refused (error 111)
I20260620 18:41:19.950055 12656 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.37.61" port: 45785 } attempt: 5
Jun 20 18:41:19 dist-test-slave-5lgp krb5kdc[12576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980879, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.62@KRBTEST.COM for kudu/127.22.37.61@KRBTEST.COM
I20260620 18:41:19.964500 12659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:19.950619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.61:45785 (local address 127.0.0.1:40206)
0620 18:41:19.950812 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:19.950832 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:19.951007 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:19.951337 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:19.951342 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:19.951856 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:41:19.952089 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:19.952101 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:19.953036 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:19.953041 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:19.954519 (+ 1478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:19.954535 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:19.955310 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:19.955316 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:19.955583 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:19.956267 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:41:19.956291 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:41:19.958676 (+ 2385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:41:19.962323 (+ 3647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:19.962327 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:19.962329 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:19.962683 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:19.963077 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:19.963080 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:19.963082 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:19.963201 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:19.963777 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:41:19.963780 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:41:19.963923 (+ 143us) client_negotiation.cc:769] Sending connection context
0620 18:41:19.964063 (+ 140us) client_negotiation.cc:240] Negotiation successful
0620 18:41:19.964252 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260620 18:41:19.964710 12734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:19.950668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40206 (local address 127.22.37.61:45785)
0620 18:41:19.950830 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:19.950834 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:19.950904 (+ 70us) server_negotiation.cc:407] Connection header received
0620 18:41:19.951120 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:19.951124 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:19.951194 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:19.951275 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:19.952225 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:19.952894 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:19.954682 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:19.955155 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:19.955260 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:19.958885 (+ 3625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:19.958891 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:19.958896 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:19.958945 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:19.962196 (+ 3251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:19.962800 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:19.962805 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:19.962809 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:19.962972 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:19.963301 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:19.963305 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:19.963308 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:19.963662 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:19.963803 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:19.964188 (+ 385us) server_negotiation.cc:299] Negotiation successful
0620 18:41:19.964406 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 18:41:19.965716 12656 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.37.60" port: 45055 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:41:19.967051 12659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:19.966479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.60:45055 (local address 127.0.0.1:46052)
0620 18:41:19.966732 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:19.966876 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.60:45055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260620 18:41:19.968096 12659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:19.967771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.60:45055 (local address 127.0.0.1:46064)
0620 18:41:19.967910 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:19.968001 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.60:45055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 18:41:19.968358 12656 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.37.60:45055: Network error: Client connection negotiation failed: client connection to 127.22.37.60:45055: connect: Connection refused (error 111)
I20260620 18:41:20.019599 12656 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.37.60" port: 45055 } attempt: 1
I20260620 18:41:20.020990 12659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:20.020374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.60:45055 (local address 127.0.0.1:46066)
0620 18:41:20.020642 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:20.020755 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.60:45055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260620 18:41:20.022176 12659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:20.021780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.60:45055 (local address 127.0.0.1:46068)
0620 18:41:20.021952 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:20.022041 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.60:45055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 18:41:20.022398 12656 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.37.60:45055: Network error: Client connection negotiation failed: client connection to 127.22.37.60:45055: connect: Connection refused (error 111)
I20260620 18:41:20.040850 12730 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.37.60" port: 45055 } attempt: 3
I20260620 18:41:20.042307 12732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:20.041664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.60:45055 (local address 127.0.0.1:46084)
0620 18:41:20.041938 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:20.042072 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.60:45055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260620 18:41:20.043498 12732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:20.042951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.60:45055 (local address 127.0.0.1:46096)
0620 18:41:20.043242 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:20.043343 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.60:45055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168}
W20260620 18:41:20.043720 12730 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.37.60:45055: Network error: Client connection negotiation failed: client connection to 127.22.37.60:45055: connect: Connection refused (error 111)
I20260620 18:41:20.090646 12656 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.37.60" port: 45055 } attempt: 2
I20260620 18:41:20.091868 12659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:20.091292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.60:45055 (local address 127.0.0.1:46106)
0620 18:41:20.091534 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:20.091660 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.60:45055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260620 18:41:20.093004 12659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:20.092563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.60:45055 (local address 127.0.0.1:46118)
0620 18:41:20.092745 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:20.092854 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.60:45055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260620 18:41:20.093218 12656 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.37.60:45055: Network error: Client connection negotiation failed: client connection to 127.22.37.60:45055: connect: Connection refused (error 111)
W20260620 18:41:20.175865 12737 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 18:41:20.176188 12737 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:41:20.176306 12737 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:41:20.176450 12737 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:41:20.183483 12737 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:41:20.183558 12737 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:41:20.183612 12737 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:41:20.183681 12737 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:41:20.183761 12737 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:41:20.186954 12730 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.37.60" port: 45055 } attempt: 4
I20260620 18:41:20.188097 12732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:20.187554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.60:45055 (local address 127.0.0.1:46120)
0620 18:41:20.187807 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:20.187917 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.60:45055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260620 18:41:20.189092 12732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:20.188719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.60:45055 (local address 127.0.0.1:46126)
0620 18:41:20.188862 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:20.188955 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.60:45055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 18:41:20.189280 12730 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.37.60:45055: Network error: Client connection negotiation failed: client connection to 127.22.37.60:45055: connect: Connection refused (error 111)
I20260620 18:41:20.194132 12737 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:33177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980527431848-22676-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980527431848-22676-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.22.37.62:38759,127.22.37.61:45785,127.22.37.60:45055
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.60.keytab
--principal=kudu/127.22.37.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.60:45055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980527431848-22676-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.22.37.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980527431848-22676-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:41:20.195333 12737 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:41:20.196561 12737 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 18:41:20.199442 12656 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.37.60" port: 45055 } attempt: 3
I20260620 18:41:20.200387 12659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:20.199922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.60:45055 (local address 127.0.0.1:46134)
0620 18:41:20.200119 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:20.200213 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.60:45055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260620 18:41:20.201452 12659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:20.200999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.60:45055 (local address 127.0.0.1:46142)
0620 18:41:20.201182 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:20.201307 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.60:45055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260620 18:41:20.201684 12656 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.37.60:45055: Network error: Client connection negotiation failed: client connection to 127.22.37.60:45055: connect: Connection refused (error 111)
W20260620 18:41:20.208459 12742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:20.208508 12743 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:20.208554 12745 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:41:20.209278 12737 server_base.cc:1034] running on GCE node
I20260620 18:41:20.210017 12737 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:41:20.211275 12737 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:41:20.212496 12737 hybrid_clock.cc:648] HybridClock initialized: now 1781980880212422 us; error 47 us; skew 500 ppm
I20260620 18:41:20.212815 12737 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:41:20 dist-test-slave-5lgp krb5kdc[12576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980880, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:41:20.216112 12737 init.cc:377] Logged in from keytab as kudu/127.22.37.60@KRBTEST.COM (short username kudu)
I20260620 18:41:20.217514 12737 webserver.cc:460] Webserver started at http://127.22.37.60:40193/ using document root <none> and password file <none>
I20260620 18:41:20.218225 12737 fs_manager.cc:359] Metadata directory not provided
I20260620 18:41:20.218345 12737 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:41:20.218645 12737 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:41:20.221580 12737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980527431848-22676-0/minicluster-data/master-2/data/instance:
uuid: "b4c4770140d0412cb916a0c38d1b015c"
format_stamp: "Formatted at 2026-06-20 18:41:20 on dist-test-slave-5lgp"
server_key: "340fcaeee21908a7a8bb0853ab0c3db0"
server_key_iv: "002d717ec8fac77a6526dd312ddc6c03"
server_key_version: "encryptionkey@0"
I20260620 18:41:20.222287 12737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980527431848-22676-0/minicluster-data/master-2/wal/instance:
uuid: "b4c4770140d0412cb916a0c38d1b015c"
format_stamp: "Formatted at 2026-06-20 18:41:20 on dist-test-slave-5lgp"
server_key: "340fcaeee21908a7a8bb0853ab0c3db0"
server_key_iv: "002d717ec8fac77a6526dd312ddc6c03"
server_key_version: "encryptionkey@0"
I20260620 18:41:20.227173 12737 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260620 18:41:20.230367 12752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:20.231609 12737 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 18:41:20.231729 12737 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980527431848-22676-0/minicluster-data/master-2/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980527431848-22676-0/minicluster-data/master-2/wal
uuid: "b4c4770140d0412cb916a0c38d1b015c"
format_stamp: "Formatted at 2026-06-20 18:41:20 on dist-test-slave-5lgp"
server_key: "340fcaeee21908a7a8bb0853ab0c3db0"
server_key_iv: "002d717ec8fac77a6526dd312ddc6c03"
server_key_version: "encryptionkey@0"
I20260620 18:41:20.231882 12737 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980527431848-22676-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980527431848-22676-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980527431848-22676-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:41:20.250519 12737 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:41:20.254683 12737 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:41:20.254930 12737 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:41:20.272722 12737 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.60:45055
I20260620 18:41:20.272742 12803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.60:45055 every 8 connection(s)
I20260620 18:41:20.274004 12737 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980527431848-22676-0/minicluster-data/master-2/data/info.pb
I20260620 18:41:20.275838 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 12737
I20260620 18:41:20.276010 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980527431848-22676-0/minicluster-data/master-2/wal/instance
I20260620 18:41:20.276429 22676 external_mini_cluster.cc:1428] Setting key 1e25e0c4c833228d829122798126179a
I20260620 18:41:20.278110 12804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:41:20.287403 12804 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.37.62" port: 38759 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:41:20 dist-test-slave-5lgp krb5kdc[12576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980878, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.37.60@KRBTEST.COM
Jun 20 18:41:20 dist-test-slave-5lgp krb5kdc[12576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980880, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.60@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM
I20260620 18:41:20.300307 12807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:20.278391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46154 (local address 127.22.37.60:45055)
0620 18:41:20.279187 (+ 796us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:20.279202 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:20.279250 (+ 48us) server_negotiation.cc:407] Connection header received
0620 18:41:20.281269 (+ 2019us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:20.281296 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:20.281819 (+ 523us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:20.282231 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:20.283403 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:20.285436 (+ 2033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:20.286660 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:20.287636 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:20.287848 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:20.292303 (+ 4455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:20.292312 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:20.292324 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:20.292362 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:20.295954 (+ 3592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:20.296727 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:20.296736 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:20.296746 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:20.296947 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:20.297306 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:20.297310 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:20.297313 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:20.298009 (+ 696us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:20.298275 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:20.298675 (+ 400us) server_negotiation.cc:299] Negotiation successful
0620 18:41:20.299096 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":305,"threads_started":1}
I20260620 18:41:20.305473 12808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:20.290112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:38759 (local address 127.0.0.1:36138)
0620 18:41:20.290962 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:20.290998 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:20.291180 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:20.291624 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:20.291631 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:20.292304 (+ 673us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:41:20.292669 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:20.292679 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:20.293731 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:20.293736 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:20.295422 (+ 1686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:20.295436 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:20.296136 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:20.296144 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:20.296267 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:20.296845 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:41:20.296871 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:41:20.299468 (+ 2597us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:41:20.302605 (+ 3137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:20.302617 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:20.302624 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:20.303174 (+ 550us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:20.303617 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:20.303621 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:20.303623 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:20.303751 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:20.304363 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:41:20.304372 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:41:20.304554 (+ 182us) client_negotiation.cc:769] Sending connection context
0620 18:41:20.304717 (+ 163us) client_negotiation.cc:240] Negotiation successful
0620 18:41:20.305004 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":388,"threads_started":1}
I20260620 18:41:20.305497 12660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:20.290102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36138 (local address 127.22.37.62:38759)
0620 18:41:20.290319 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:20.290327 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:20.291102 (+ 775us) server_negotiation.cc:407] Connection header received
0620 18:41:20.291328 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:20.291335 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:20.291417 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:20.291562 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:20.292818 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:20.293574 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:20.295557 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:20.295959 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:20.296099 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:20.299656 (+ 3557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:20.299661 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:20.299664 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:20.299694 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:20.302486 (+ 2792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:20.303279 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:20.303284 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:20.303288 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:20.303496 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:20.303844 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:20.303848 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:20.303851 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:20.304235 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:20.304404 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:20.304834 (+ 430us) server_negotiation.cc:299] Negotiation successful
0620 18:41:20.305045 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 18:41:20.306826 12804 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.37.61" port: 45785 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:41:20 dist-test-slave-5lgp krb5kdc[12576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980878, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM
Jun 20 18:41:20 dist-test-slave-5lgp krb5kdc[12576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980880, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.60@KRBTEST.COM for kudu/127.22.37.61@KRBTEST.COM
I20260620 18:41:20.319111 12809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:20.303131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36146 (local address 127.22.37.62:38759)
0620 18:41:20.303750 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:20.303753 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:20.303770 (+ 17us) server_negotiation.cc:407] Connection header received
0620 18:41:20.303828 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:20.303834 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:20.303900 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:20.304002 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:20.305077 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:20.306313 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:20.307786 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:20.308193 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:20.308329 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:20.312630 (+ 4301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:20.312634 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:20.312637 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:20.312665 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:20.316209 (+ 3544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:20.316979 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:20.316984 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:20.316987 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:20.317122 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:20.317496 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:20.317501 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:20.317505 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:20.317891 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:20.318181 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:20.318361 (+ 180us) server_negotiation.cc:299] Negotiation successful
0620 18:41:20.318669 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":499,"thread_start_us":302,"threads_started":1}
I20260620 18:41:20.321115 12808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:20.307600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.61:45785 (local address 127.0.0.1:40212)
0620 18:41:20.307945 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:20.307959 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:20.308078 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:20.308409 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:20.308413 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:20.308861 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:41:20.309066 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:20.309077 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:20.310092 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:20.310099 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:20.311031 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:20.311045 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:20.311808 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:20.311817 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:20.312037 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:20.312558 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:41:20.312583 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:41:20.314608 (+ 2025us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:41:20.318401 (+ 3793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:20.318436 (+ 35us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:20.318440 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:20.318917 (+ 477us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:20.319352 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:20.319357 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:20.319360 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:20.319521 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:20.320164 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:41:20.320168 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:41:20.320326 (+ 158us) client_negotiation.cc:769] Sending connection context
0620 18:41:20.320474 (+ 148us) client_negotiation.cc:240] Negotiation successful
0620 18:41:20.320780 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85}
I20260620 18:41:20.321527 12734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:20.307709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40212 (local address 127.22.37.61:45785)
0620 18:41:20.307871 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:20.307876 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:20.308028 (+ 152us) server_negotiation.cc:407] Connection header received
0620 18:41:20.308211 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:20.308216 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:20.308290 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:20.308391 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:20.309216 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:20.309966 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:20.311172 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:20.311658 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:20.311753 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:20.314799 (+ 3046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:20.314805 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:20.314811 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:20.314855 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:20.318263 (+ 3408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:20.319045 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:20.319052 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:20.319055 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:20.319216 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:20.319661 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:20.319667 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:20.319672 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:20.320044 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:20.320196 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:20.320618 (+ 422us) server_negotiation.cc:299] Negotiation successful
0620 18:41:20.320857 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 18:41:20.322602 12804 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.37.60" port: 45055 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:41:20 dist-test-slave-5lgp krb5kdc[12576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980880, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.60@KRBTEST.COM for kudu/127.22.37.60@KRBTEST.COM
Jun 20 18:41:20 dist-test-slave-5lgp krb5kdc[12576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980878, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.37.61@KRBTEST.COM
I20260620 18:41:20.337294 12808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:20.323323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.60:45055 (local address 127.0.0.1:46156)
0620 18:41:20.323497 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:20.323516 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:20.323612 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:20.323980 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:20.323984 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:20.324433 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:41:20.324599 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:20.324612 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:20.325601 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:20.325609 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:20.326544 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:20.326557 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:20.327764 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:20.327771 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:20.328004 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:20.328597 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:41:20.328624 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:41:20.330693 (+ 2069us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:41:20.334050 (+ 3357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:20.334058 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:20.334065 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:20.334751 (+ 686us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:20.335453 (+ 702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:20.335460 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:20.335466 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:20.335698 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:20.336284 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:41:20.336288 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:41:20.336415 (+ 127us) client_negotiation.cc:769] Sending connection context
0620 18:41:20.336554 (+ 139us) client_negotiation.cc:240] Negotiation successful
0620 18:41:20.336866 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260620 18:41:20.337510 12807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:20.323618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46156 (local address 127.22.37.60:45055)
0620 18:41:20.323781 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:20.323785 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:20.323796 (+ 11us) server_negotiation.cc:407] Connection header received
0620 18:41:20.323826 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:20.323829 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:20.323872 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:20.323968 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:20.324762 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:20.325494 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:20.326692 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:20.327564 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:20.327726 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:20.330876 (+ 3150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:20.330880 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:20.330883 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:20.330915 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:20.333848 (+ 2933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:20.335159 (+ 1311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:20.335164 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:20.335167 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:20.335316 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:20.335831 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:20.335837 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:20.335842 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:20.336180 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:20.336310 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:20.336652 (+ 342us) server_negotiation.cc:299] Negotiation successful
0620 18:41:20.337032 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":147,"server-negotiator.queue_time_us":77}
I20260620 18:41:20.339941 12734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:20.320972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40222 (local address 127.22.37.61:45785)
0620 18:41:20.322023 (+ 1051us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:20.322026 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:20.322038 (+ 12us) server_negotiation.cc:407] Connection header received
0620 18:41:20.322070 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:20.322073 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:20.322117 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:20.322233 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:20.324689 (+ 2456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:20.326163 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:20.327466 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:20.328052 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:20.328191 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:20.334292 (+ 6101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:20.334297 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:20.334299 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:20.334327 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:20.337202 (+ 2875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:20.337979 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:20.337991 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:20.337993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:20.338121 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:20.338495 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:20.338501 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:20.338504 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:20.338854 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:20.339143 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:20.339329 (+ 186us) server_negotiation.cc:299] Negotiation successful
0620 18:41:20.339543 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":979,"thread_start_us":15130,"threads_started":1}
I20260620 18:41:20.343041 12804 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4c4770140d0412cb916a0c38d1b015c: Bootstrap starting.
I20260620 18:41:20.344918 22676 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:41:20.346458 12804 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b4c4770140d0412cb916a0c38d1b015c: Neither blocks nor log segments found. Creating new log.
I20260620 18:41:20.347592 12804 log.cc:824] T 00000000000000000000000000000000 P b4c4770140d0412cb916a0c38d1b015c: Log is configured to *not* fsync() on all Append() calls
I20260620 18:41:20.350708 12804 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4c4770140d0412cb916a0c38d1b015c: No bootstrap required, opened a new log
I20260620 18:41:20.347600 12631 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "45e9b431d78d454da7c9418b68be52c9"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36146
I20260620 18:41:20.361676 12804 raft_consensus.cc:348] T 00000000000000000000000000000000 P b4c4770140d0412cb916a0c38d1b015c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45e9b431d78d454da7c9418b68be52c9" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 38759 } } peers { permanent_uuid: "58601e366a404d81859c358d347cf516" member_type: VOTER last_known_addr { host: "127.22.37.61" port: 45785 } } peers { permanent_uuid: "b4c4770140d0412cb916a0c38d1b015c" member_type: VOTER last_known_addr { host: "127.22.37.60" port: 45055 } }
I20260620 18:41:20.362089 12804 raft_consensus.cc:729] T 00000000000000000000000000000000 P b4c4770140d0412cb916a0c38d1b015c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4c4770140d0412cb916a0c38d1b015c, State: Initialized, Role: FOLLOWER
I20260620 18:41:20.362612 12804 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4c4770140d0412cb916a0c38d1b015c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45e9b431d78d454da7c9418b68be52c9" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 38759 } } peers { permanent_uuid: "58601e366a404d81859c358d347cf516" member_type: VOTER last_known_addr { host: "127.22.37.61" port: 45785 } } peers { permanent_uuid: "b4c4770140d0412cb916a0c38d1b015c" member_type: VOTER last_known_addr { host: "127.22.37.60" port: 45055 } }
I20260620 18:41:20.363521 12812 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4c4770140d0412cb916a0c38d1b015c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45e9b431d78d454da7c9418b68be52c9" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 38759 } } peers { permanent_uuid: "58601e366a404d81859c358d347cf516" member_type: VOTER last_known_addr { host: "127.22.37.61" port: 45785 } } peers { permanent_uuid: "b4c4770140d0412cb916a0c38d1b015c" member_type: VOTER last_known_addr { host: "127.22.37.60" port: 45055 } } }
I20260620 18:41:20.363847 12812 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4c4770140d0412cb916a0c38d1b015c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:41:20.364735 12804 sys_catalog.cc:564] T 00000000000000000000000000000000 P b4c4770140d0412cb916a0c38d1b015c [sys.catalog]: configured and running, proceeding with master startup.
W20260620 18:41:20.374418 12823 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b4c4770140d0412cb916a0c38d1b015c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 18:41:20.374562 12823 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 18:41:20.378881 12656 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.37.60" port: 45055 } attempt: 4
I20260620 18:41:20.372327 12737 master_runner.cc:186] Error getting master registration for 127.22.37.62:38759: OK, instance_id { permanent_uuid: "45e9b431d78d454da7c9418b68be52c9" instance_seqno: 1781980879238432 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 18:41:20.383425 12737 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Jun 20 18:41:20 dist-test-slave-5lgp krb5kdc[12576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980879, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.62@KRBTEST.COM for kudu/127.22.37.60@KRBTEST.COM
I20260620 18:41:20.397815 12807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:20.379733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46160 (local address 127.22.37.60:45055)
0620 18:41:20.379935 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:20.379945 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:20.379959 (+ 14us) server_negotiation.cc:407] Connection header received
0620 18:41:20.380005 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:20.380008 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:20.380069 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:20.380182 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:20.381104 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:20.381918 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:20.383618 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:20.384065 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:20.384198 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:20.391523 (+ 7325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:20.391528 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:20.391531 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:20.391562 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:20.395118 (+ 3556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:20.395770 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:20.395776 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:20.395780 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:20.395910 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:20.396287 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:20.396293 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:20.396296 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:20.396652 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:20.396830 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:20.397200 (+ 370us) server_negotiation.cc:299] Negotiation successful
0620 18:41:20.397443 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 18:41:20.397810 12659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:20.379458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.60:45055 (local address 127.0.0.1:46160)
0620 18:41:20.379646 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:20.379665 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:20.379776 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:20.380210 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:20.380214 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:20.380741 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:41:20.380952 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:20.380965 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:20.382049 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:20.382056 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:20.383500 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:20.383515 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:20.384209 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:20.384215 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:20.384414 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:20.388772 (+ 4358us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:41:20.388790 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:41:20.391327 (+ 2537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:41:20.395262 (+ 3935us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:20.395267 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:20.395271 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:20.395644 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:20.396022 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:20.396030 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:20.396034 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:20.396178 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:20.396773 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:41:20.396778 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:41:20.396957 (+ 179us) client_negotiation.cc:769] Sending connection context
0620 18:41:20.397104 (+ 147us) client_negotiation.cc:240] Negotiation successful
0620 18:41:20.397420 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260620 18:41:20.402582 12656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45e9b431d78d454da7c9418b68be52c9: Bootstrap starting.
I20260620 18:41:20.405862 12656 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 45e9b431d78d454da7c9418b68be52c9: Neither blocks nor log segments found. Creating new log.
I20260620 18:41:20.406854 12656 log.cc:824] T 00000000000000000000000000000000 P 45e9b431d78d454da7c9418b68be52c9: Log is configured to *not* fsync() on all Append() calls
I20260620 18:41:20.409726 12656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45e9b431d78d454da7c9418b68be52c9: No bootstrap required, opened a new log
I20260620 18:41:20.414710 12656 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45e9b431d78d454da7c9418b68be52c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45e9b431d78d454da7c9418b68be52c9" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 38759 } } peers { permanent_uuid: "58601e366a404d81859c358d347cf516" member_type: VOTER last_known_addr { host: "127.22.37.61" port: 45785 } } peers { permanent_uuid: "b4c4770140d0412cb916a0c38d1b015c" member_type: VOTER last_known_addr { host: "127.22.37.60" port: 45055 } }
I20260620 18:41:20.415020 12656 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45e9b431d78d454da7c9418b68be52c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45e9b431d78d454da7c9418b68be52c9, State: Initialized, Role: FOLLOWER
I20260620 18:41:20.415377 12656 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45e9b431d78d454da7c9418b68be52c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45e9b431d78d454da7c9418b68be52c9" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 38759 } } peers { permanent_uuid: "58601e366a404d81859c358d347cf516" member_type: VOTER last_known_addr { host: "127.22.37.61" port: 45785 } } peers { permanent_uuid: "b4c4770140d0412cb916a0c38d1b015c" member_type: VOTER last_known_addr { host: "127.22.37.60" port: 45055 } }
I20260620 18:41:20.416247 12826 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45e9b431d78d454da7c9418b68be52c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45e9b431d78d454da7c9418b68be52c9" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 38759 } } peers { permanent_uuid: "58601e366a404d81859c358d347cf516" member_type: VOTER last_known_addr { host: "127.22.37.61" port: 45785 } } peers { permanent_uuid: "b4c4770140d0412cb916a0c38d1b015c" member_type: VOTER last_known_addr { host: "127.22.37.60" port: 45055 } } }
I20260620 18:41:20.416621 12826 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45e9b431d78d454da7c9418b68be52c9 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:41:20.417447 12656 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45e9b431d78d454da7c9418b68be52c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:41:20.417586 12631 raft_consensus.cc:482] T 00000000000000000000000000000000 P 45e9b431d78d454da7c9418b68be52c9 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 18:41:20.417882 12631 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45e9b431d78d454da7c9418b68be52c9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:41:20.421607 12631 raft_consensus.cc:504] T 00000000000000000000000000000000 P 45e9b431d78d454da7c9418b68be52c9 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45e9b431d78d454da7c9418b68be52c9" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 38759 } } peers { permanent_uuid: "58601e366a404d81859c358d347cf516" member_type: VOTER last_known_addr { host: "127.22.37.61" port: 45785 } } peers { permanent_uuid: "b4c4770140d0412cb916a0c38d1b015c" member_type: VOTER last_known_addr { host: "127.22.37.60" port: 45055 } }
I20260620 18:41:20.424296 12631 leader_election.cc:288] T 00000000000000000000000000000000 P 45e9b431d78d454da7c9418b68be52c9 [CANDIDATE]: Term 1 election: Requested vote from peers 58601e366a404d81859c358d347cf516 (127.22.37.61:45785), b4c4770140d0412cb916a0c38d1b015c (127.22.37.60:45055)
I20260620 18:41:20.425776 12779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "45e9b431d78d454da7c9418b68be52c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b4c4770140d0412cb916a0c38d1b015c"
I20260620 18:41:20.426355 12779 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b4c4770140d0412cb916a0c38d1b015c [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:41:20.432379 12779 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b4c4770140d0412cb916a0c38d1b015c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45e9b431d78d454da7c9418b68be52c9 in term 1.
I20260620 18:41:20.433427 12607 leader_election.cc:302] T 00000000000000000000000000000000 P 45e9b431d78d454da7c9418b68be52c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45e9b431d78d454da7c9418b68be52c9, b4c4770140d0412cb916a0c38d1b015c; no voters:
I20260620 18:41:20.434041 12826 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 45e9b431d78d454da7c9418b68be52c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:41:20.435572 12826 raft_consensus.cc:686] T 00000000000000000000000000000000 P 45e9b431d78d454da7c9418b68be52c9 [term 1 LEADER]: Becoming Leader. State: Replica: 45e9b431d78d454da7c9418b68be52c9, State: Running, Role: LEADER
I20260620 18:41:20.436316 12826 consensus_queue.cc:237] T 00000000000000000000000000000000 P 45e9b431d78d454da7c9418b68be52c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45e9b431d78d454da7c9418b68be52c9" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 38759 } } peers { permanent_uuid: "58601e366a404d81859c358d347cf516" member_type: VOTER last_known_addr { host: "127.22.37.61" port: 45785 } } peers { permanent_uuid: "b4c4770140d0412cb916a0c38d1b015c" member_type: VOTER last_known_addr { host: "127.22.37.60" port: 45055 } }
I20260620 18:41:20.442154 12839 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45e9b431d78d454da7c9418b68be52c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45e9b431d78d454da7c9418b68be52c9. Latest consensus state: current_term: 1 leader_uuid: "45e9b431d78d454da7c9418b68be52c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45e9b431d78d454da7c9418b68be52c9" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 38759 } } peers { permanent_uuid: "58601e366a404d81859c358d347cf516" member_type: VOTER last_known_addr { host: "127.22.37.61" port: 45785 } } peers { permanent_uuid: "b4c4770140d0412cb916a0c38d1b015c" member_type: VOTER last_known_addr { host: "127.22.37.60" port: 45055 } } }
I20260620 18:41:20.442703 12839 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45e9b431d78d454da7c9418b68be52c9 [sys.catalog]: This master's current role is: LEADER
I20260620 18:41:20.445403 12589 master_runner.cc:186] Error getting master registration for 127.22.37.61:45785: OK, instance_id { permanent_uuid: "58601e366a404d81859c358d347cf516" instance_seqno: 1781980879757521 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 18:41:20.468566 12589 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260620 18:41:20.477835 12610 tablet.cc:2369] T 00000000000000000000000000000000 P 45e9b431d78d454da7c9418b68be52c9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:41:20.480306 12837 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 45e9b431d78d454da7c9418b68be52c9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 18:41:20.480449 12837 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 18:41:20.480629 12841 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:41:20.482837 12841 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:41:20.488466 12730 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.37.60" port: 45055 } attempt: 5
Jun 20 18:41:20 dist-test-slave-5lgp krb5kdc[12576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980879, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.61@KRBTEST.COM for kudu/127.22.37.60@KRBTEST.COM
W20260620 18:41:20.507019 12758 tablet.cc:2369] T 00000000000000000000000000000000 P b4c4770140d0412cb916a0c38d1b015c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:41:20.509483 12732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:20.489129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.60:45055 (local address 127.0.0.1:46170)
0620 18:41:20.489450 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:20.489474 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:20.489619 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:20.490106 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:20.490114 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:20.490805 (+ 691us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:41:20.491079 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:20.491095 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:20.493072 (+ 1977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:20.493081 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:20.494824 (+ 1743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:20.494839 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:20.496035 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:20.496042 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:20.496249 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:20.496928 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:41:20.496953 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:41:20.500666 (+ 3713us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:41:20.504827 (+ 4161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:20.504832 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:20.504837 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:20.505360 (+ 523us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:20.506008 (+ 648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:20.506015 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:20.506022 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:20.506236 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:20.508433 (+ 2197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:41:20.508440 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:41:20.508591 (+ 151us) client_negotiation.cc:769] Sending connection context
0620 18:41:20.508746 (+ 155us) client_negotiation.cc:240] Negotiation successful
0620 18:41:20.509047 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171}
I20260620 18:41:20.510602 12807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:20.489603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46170 (local address 127.22.37.60:45055)
0620 18:41:20.489768 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:20.489773 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:20.489792 (+ 19us) server_negotiation.cc:407] Connection header received
0620 18:41:20.489862 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:20.489868 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:20.489962 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:20.490072 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:20.491312 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:20.492852 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:20.494998 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:20.495704 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:20.495834 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:20.500887 (+ 5053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:20.500894 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:20.500899 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:20.500940 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:20.504641 (+ 3701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:20.505554 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:20.505562 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:20.505569 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:20.505828 (+ 259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:20.506380 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:20.506387 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:20.506394 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:20.508210 (+ 1816us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:20.508365 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:20.509980 (+ 1615us) server_negotiation.cc:299] Negotiation successful
0620 18:41:20.510215 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 18:41:20.517061 12730 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58601e366a404d81859c358d347cf516: Bootstrap starting.
I20260620 18:41:20.521484 12730 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 58601e366a404d81859c358d347cf516: Neither blocks nor log segments found. Creating new log.
I20260620 18:41:20.523168 12730 log.cc:824] T 00000000000000000000000000000000 P 58601e366a404d81859c358d347cf516: Log is configured to *not* fsync() on all Append() calls
I20260620 18:41:20.531296 12730 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58601e366a404d81859c358d347cf516: No bootstrap required, opened a new log
I20260620 18:41:20.544329 12809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:20.488460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36150 (local address 127.22.37.62:38759)
0620 18:41:20.488644 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:20.488651 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:20.508747 (+ 20096us) server_negotiation.cc:407] Connection header received
0620 18:41:20.509270 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:20.509276 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:20.509403 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:20.509481 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:20.517496 (+ 8015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:20.518526 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:20.520552 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:20.521178 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:20.521312 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:20.526736 (+ 5424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:20.526746 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:20.526754 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:20.526809 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:20.530728 (+ 3919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:20.541519 (+ 10791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:20.541531 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:20.541535 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:20.541743 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:20.542340 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:20.542345 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:20.542348 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:20.542738 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:20.542968 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:20.543689 (+ 721us) server_negotiation.cc:299] Negotiation successful
0620 18:41:20.543961 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 18:41:20.545848 12779 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b4c4770140d0412cb916a0c38d1b015c [term 1 FOLLOWER]: Refusing update from remote peer 45e9b431d78d454da7c9418b68be52c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:41:20.546846 12839 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 45e9b431d78d454da7c9418b68be52c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4c4770140d0412cb916a0c38d1b015c" member_type: VOTER last_known_addr { host: "127.22.37.60" port: 45055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:41:20.584268 12807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:20.543993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46182 (local address 127.22.37.60:45055)
0620 18:41:20.544182 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:20.544188 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:20.546966 (+ 2778us) server_negotiation.cc:407] Connection header received
0620 18:41:20.547365 (+ 399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:20.547370 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:20.547479 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:20.547553 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:20.549540 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:20.550586 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:20.552616 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:20.553295 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:20.553412 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:20.557360 (+ 3948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:20.557381 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:20.557387 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:20.557438 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:20.561529 (+ 4091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:20.565715 (+ 4186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:20.565724 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:20.565730 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:20.575893 (+ 10163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:20.581855 (+ 5962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:20.581865 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:20.581871 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:20.582355 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:20.582666 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:20.583401 (+ 735us) server_negotiation.cc:299] Negotiation successful
0620 18:41:20.583748 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 18:41:20.596098 12734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:20.489187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40234 (local address 127.22.37.61:45785)
0620 18:41:20.489510 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:20.489516 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:20.543615 (+ 54099us) server_negotiation.cc:407] Connection header received
0620 18:41:20.554073 (+ 10458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:20.554078 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:20.554177 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:20.554254 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:20.564044 (+ 9790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:20.565146 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:20.575694 (+ 10548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:20.576363 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:20.576495 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:20.587152 (+ 10657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:20.587159 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:20.587163 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:20.587209 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:20.591109 (+ 3900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:20.592616 (+ 1507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:20.592623 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:20.592626 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:20.592927 (+ 301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:20.593767 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:20.593773 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:20.593777 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:20.594162 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:20.594415 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:20.595461 (+ 1046us) server_negotiation.cc:299] Negotiation successful
0620 18:41:20.595702 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213}
I20260620 18:41:20.596410 12812 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4c4770140d0412cb916a0c38d1b015c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45e9b431d78d454da7c9418b68be52c9. Latest consensus state: current_term: 1 leader_uuid: "45e9b431d78d454da7c9418b68be52c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45e9b431d78d454da7c9418b68be52c9" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 38759 } } peers { permanent_uuid: "58601e366a404d81859c358d347cf516" member_type: VOTER last_known_addr { host: "127.22.37.61" port: 45785 } } peers { permanent_uuid: "b4c4770140d0412cb916a0c38d1b015c" member_type: VOTER last_known_addr { host: "127.22.37.60" port: 45055 } } }
I20260620 18:41:20.573154 12730 raft_consensus.cc:348] T 00000000000000000000000000000000 P 58601e366a404d81859c358d347cf516 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45e9b431d78d454da7c9418b68be52c9" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 38759 } } peers { permanent_uuid: "58601e366a404d81859c358d347cf516" member_type: VOTER last_known_addr { host: "127.22.37.61" port: 45785 } } peers { permanent_uuid: "b4c4770140d0412cb916a0c38d1b015c" member_type: VOTER last_known_addr { host: "127.22.37.60" port: 45055 } }
I20260620 18:41:20.596745 12812 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4c4770140d0412cb916a0c38d1b015c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:41:20.596899 12730 raft_consensus.cc:729] T 00000000000000000000000000000000 P 58601e366a404d81859c358d347cf516 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58601e366a404d81859c358d347cf516, State: Initialized, Role: FOLLOWER
I20260620 18:41:20.597582 12730 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58601e366a404d81859c358d347cf516 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45e9b431d78d454da7c9418b68be52c9" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 38759 } } peers { permanent_uuid: "58601e366a404d81859c358d347cf516" member_type: VOTER last_known_addr { host: "127.22.37.61" port: 45785 } } peers { permanent_uuid: "b4c4770140d0412cb916a0c38d1b015c" member_type: VOTER last_known_addr { host: "127.22.37.60" port: 45055 } }
I20260620 18:41:20.600343 12839 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45e9b431d78d454da7c9418b68be52c9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "45e9b431d78d454da7c9418b68be52c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45e9b431d78d454da7c9418b68be52c9" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 38759 } } peers { permanent_uuid: "58601e366a404d81859c358d347cf516" member_type: VOTER last_known_addr { host: "127.22.37.61" port: 45785 } } peers { permanent_uuid: "b4c4770140d0412cb916a0c38d1b015c" member_type: VOTER last_known_addr { host: "127.22.37.60" port: 45055 } } }
I20260620 18:41:20.600651 12839 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45e9b431d78d454da7c9418b68be52c9 [sys.catalog]: This master's current role is: LEADER
I20260620 18:41:20.605346 12841 catalog_manager.cc:1347] Generated new cluster ID: cc9ca0ac29dc4b22abb9b8d5301b3b7a
I20260620 18:41:20.605468 12841 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:41:20.608423 12853 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58601e366a404d81859c358d347cf516 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45e9b431d78d454da7c9418b68be52c9" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 38759 } } peers { permanent_uuid: "58601e366a404d81859c358d347cf516" member_type: VOTER last_known_addr { host: "127.22.37.61" port: 45785 } } peers { permanent_uuid: "b4c4770140d0412cb916a0c38d1b015c" member_type: VOTER last_known_addr { host: "127.22.37.60" port: 45055 } } }
I20260620 18:41:20.610474 12730 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58601e366a404d81859c358d347cf516 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:41:20.610854 12704 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 58601e366a404d81859c358d347cf516 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:41:20.614610 12704 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 58601e366a404d81859c358d347cf516 [term 1 FOLLOWER]: Refusing update from remote peer 45e9b431d78d454da7c9418b68be52c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:41:20.615162 12705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "45e9b431d78d454da7c9418b68be52c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "58601e366a404d81859c358d347cf516"
I20260620 18:41:20.615756 12826 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 45e9b431d78d454da7c9418b68be52c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58601e366a404d81859c358d347cf516" member_type: VOTER last_known_addr { host: "127.22.37.61" port: 45785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:41:20.618742 12705 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 58601e366a404d81859c358d347cf516 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45e9b431d78d454da7c9418b68be52c9 in term 1.
I20260620 18:41:20.610234 12853 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58601e366a404d81859c358d347cf516 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:41:20.620883 12812 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4c4770140d0412cb916a0c38d1b015c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "45e9b431d78d454da7c9418b68be52c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45e9b431d78d454da7c9418b68be52c9" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 38759 } } peers { permanent_uuid: "58601e366a404d81859c358d347cf516" member_type: VOTER last_known_addr { host: "127.22.37.61" port: 45785 } } peers { permanent_uuid: "b4c4770140d0412cb916a0c38d1b015c" member_type: VOTER last_known_addr { host: "127.22.37.60" port: 45055 } } }
I20260620 18:41:20.621222 12812 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4c4770140d0412cb916a0c38d1b015c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:41:20.634919 12841 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:41:20.637149 12841 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:41:20.663455 12859 mvcc.cc:205] Tried to move back new op lower bound from 7298993686699020288 to 7298993686274084864. Current Snapshot: MvccSnapshot[applied={T|T < 7298993686699020288}]
I20260620 18:41:20.674400 12853 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58601e366a404d81859c358d347cf516 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45e9b431d78d454da7c9418b68be52c9. Latest consensus state: current_term: 1 leader_uuid: "45e9b431d78d454da7c9418b68be52c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45e9b431d78d454da7c9418b68be52c9" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 38759 } } peers { permanent_uuid: "58601e366a404d81859c358d347cf516" member_type: VOTER last_known_addr { host: "127.22.37.61" port: 45785 } } peers { permanent_uuid: "b4c4770140d0412cb916a0c38d1b015c" member_type: VOTER last_known_addr { host: "127.22.37.60" port: 45055 } } }
I20260620 18:41:20.674722 12853 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58601e366a404d81859c358d347cf516 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:41:20.675192 12857 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58601e366a404d81859c358d347cf516 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "45e9b431d78d454da7c9418b68be52c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45e9b431d78d454da7c9418b68be52c9" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 38759 } } peers { permanent_uuid: "58601e366a404d81859c358d347cf516" member_type: VOTER last_known_addr { host: "127.22.37.61" port: 45785 } } peers { permanent_uuid: "b4c4770140d0412cb916a0c38d1b015c" member_type: VOTER last_known_addr { host: "127.22.37.60" port: 45055 } } }
I20260620 18:41:20.675546 12857 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58601e366a404d81859c358d347cf516 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:41:20.676213 12841 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 45e9b431d78d454da7c9418b68be52c9: Generated new TSK 0
I20260620 18:41:20.676924 12826 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45e9b431d78d454da7c9418b68be52c9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "45e9b431d78d454da7c9418b68be52c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45e9b431d78d454da7c9418b68be52c9" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 38759 } } peers { permanent_uuid: "58601e366a404d81859c358d347cf516" member_type: VOTER last_known_addr { host: "127.22.37.61" port: 45785 } } peers { permanent_uuid: "b4c4770140d0412cb916a0c38d1b015c" member_type: VOTER last_known_addr { host: "127.22.37.60" port: 45055 } } }
I20260620 18:41:20.677206 12826 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45e9b431d78d454da7c9418b68be52c9 [sys.catalog]: This master's current role is: LEADER
I20260620 18:41:20.677562 12841 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:41:20.715054 12871 catalog_manager.cc:1259] Loaded cluster ID: cc9ca0ac29dc4b22abb9b8d5301b3b7a
I20260620 18:41:20.715255 12871 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 58601e366a404d81859c358d347cf516: loading cluster ID for follower catalog manager: success
I20260620 18:41:20.719250 12871 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 58601e366a404d81859c358d347cf516: acquiring CA information for follower catalog manager: success
I20260620 18:41:20.722074 12871 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 58601e366a404d81859c358d347cf516: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 18:41:20.833195 12809 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:41:20.806964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36160 (local address 127.22.37.62:38759)
0620 18:41:20.807151 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:20.807158 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:20.828585 (+ 21427us) server_negotiation.cc:407] Connection header received
0620 18:41:20.831349 (+ 2764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:20.831357 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:20.831461 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:20.831567 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:20.832884 (+ 1317us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36160: BlockingRecv error: recv got EOF from 127.0.0.1:36160 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260620 18:41:20.838241 12734 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:41:20.813625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40244 (local address 127.22.37.61:45785)
0620 18:41:20.813861 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:20.813867 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:20.834519 (+ 20652us) server_negotiation.cc:407] Connection header received
0620 18:41:20.835080 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:20.835086 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:20.835186 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:20.835263 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:20.837969 (+ 2706us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40244: BlockingRecv error: recv got EOF from 127.0.0.1:40244 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260620 18:41:20.839628 12807 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:41:20.829208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46188 (local address 127.22.37.60:45055)
0620 18:41:20.829380 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:20.829387 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:20.836473 (+ 7086us) server_negotiation.cc:407] Connection header received
0620 18:41:20.836892 (+ 419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:20.836899 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:20.837007 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:20.837078 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:20.839372 (+ 2294us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46188: BlockingRecv error: recv got EOF from 127.0.0.1:46188 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 18:41:20.888713 12734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:20.872105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40258 (local address 127.22.37.61:45785)
0620 18:41:20.872310 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:20.872316 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:20.880599 (+ 8283us) server_negotiation.cc:407] Connection header received
0620 18:41:20.881179 (+ 580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:20.881187 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:20.881456 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:20.881607 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:41:20.882419 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:20.883608 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:20.885602 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:20.886245 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:20.886354 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:20.887077 (+ 723us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:41:20.887831 (+ 754us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:41:20.887966 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:20.888225 (+ 259us) server_negotiation.cc:299] Negotiation successful
0620 18:41:20.888346 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260620 18:41:20.933252 12807 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:41:20.923297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46198 (local address 127.22.37.60:45055)
0620 18:41:20.923521 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:20.923529 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:20.930871 (+ 7342us) server_negotiation.cc:407] Connection header received
0620 18:41:20.931477 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:20.931485 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:20.931650 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:20.931754 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:20.932962 (+ 1208us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46198: BlockingRecv error: recv got EOF from 127.0.0.1:46198 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 18:41:20.980700 12734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:20.964962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40270 (local address 127.22.37.61:45785)
0620 18:41:20.965209 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:20.965235 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:20.972848 (+ 7613us) server_negotiation.cc:407] Connection header received
0620 18:41:20.973492 (+ 644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:20.973502 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:20.973598 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:20.973691 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:41:20.974457 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:20.975628 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:20.977759 (+ 2131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:20.978512 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:20.978617 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:20.979364 (+ 747us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:41:20.979538 (+ 174us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:41:20.979611 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:20.980238 (+ 627us) server_negotiation.cc:299] Negotiation successful
0620 18:41:20.980345 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260620 18:41:21.024605 12807 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:41:21.015057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46208 (local address 127.22.37.60:45055)
0620 18:41:21.015248 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:21.015255 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:21.022765 (+ 7510us) server_negotiation.cc:407] Connection header received
0620 18:41:21.023244 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:21.023251 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:21.023352 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:21.023422 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:21.024376 (+ 954us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46208: BlockingRecv error: recv got EOF from 127.0.0.1:46208 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 18:41:21.077706 12734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:21.061667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40276 (local address 127.22.37.61:45785)
0620 18:41:21.061873 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:21.061880 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:21.069516 (+ 7636us) server_negotiation.cc:407] Connection header received
0620 18:41:21.070019 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:21.070027 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:21.070127 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:21.070191 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:41:21.070869 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:21.072056 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:21.075047 (+ 2991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:21.075714 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:21.075800 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:21.076833 (+ 1033us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:41:21.076975 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:41:21.077031 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:21.077241 (+ 210us) server_negotiation.cc:299] Negotiation successful
0620 18:41:21.077335 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260620 18:41:21.101361 12807 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:41:21.099122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46210 (local address 127.22.37.60:45055)
0620 18:41:21.099377 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:21.099386 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:21.099900 (+ 514us) server_negotiation.cc:407] Connection header received
0620 18:41:21.100132 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:21.100138 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:21.100225 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:21.100337 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:21.101110 (+ 773us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46210: BlockingRecv error: recv got EOF from 127.0.0.1:46210 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 18:41:21.114804 12734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:21.108801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40292 (local address 127.22.37.61:45785)
0620 18:41:21.109095 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:21.109104 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:21.109640 (+ 536us) server_negotiation.cc:407] Connection header received
0620 18:41:21.109847 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:21.109852 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:21.109956 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:21.110063 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:41:21.110487 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:21.111395 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:21.112984 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:21.113521 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:21.113670 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:21.114065 (+ 395us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:41:21.114172 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:41:21.114269 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:21.114415 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 18:41:21.114494 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138}
W20260620 18:41:21.129662 12807 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:41:21.121164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46220 (local address 127.22.37.60:45055)
0620 18:41:21.121489 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:21.121496 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:21.128324 (+ 6828us) server_negotiation.cc:407] Connection header received
0620 18:41:21.128529 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:21.128535 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:21.128626 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:21.128736 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:21.129484 (+ 748us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46220: BlockingRecv error: recv got EOF from 127.0.0.1:46220 (error 108)
Metrics: {"server-negotiator.queue_time_us":155}
I20260620 18:41:21.153542 12734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:21.147885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40296 (local address 127.22.37.61:45785)
0620 18:41:21.148139 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:21.148148 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:21.148717 (+ 569us) server_negotiation.cc:407] Connection header received
0620 18:41:21.148954 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:21.148960 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:21.149040 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:21.149146 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:41:21.149613 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:21.150366 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:21.151694 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:21.152185 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:21.152326 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:21.152716 (+ 390us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:41:21.152835 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:41:21.152940 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:21.153085 (+ 145us) server_negotiation.cc:299] Negotiation successful
0620 18:41:21.153163 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260620 18:41:21.173745 12807 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:41:21.163122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46224 (local address 127.22.37.60:45055)
0620 18:41:21.163334 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:21.163342 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:21.172214 (+ 8872us) server_negotiation.cc:407] Connection header received
0620 18:41:21.172468 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:21.172473 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:21.172548 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:21.172654 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:21.173495 (+ 841us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46224: BlockingRecv error: recv got EOF from 127.0.0.1:46224 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 18:41:21.199277 12734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:21.193391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40308 (local address 127.22.37.61:45785)
0620 18:41:21.193747 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:21.193757 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:21.194410 (+ 653us) server_negotiation.cc:407] Connection header received
0620 18:41:21.194635 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:21.194641 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:21.194722 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:21.194822 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:41:21.195247 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:21.196060 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:21.197400 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:21.197970 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:21.198120 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:21.198539 (+ 419us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:41:21.198641 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:41:21.198740 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:21.198887 (+ 147us) server_negotiation.cc:299] Negotiation successful
0620 18:41:21.198972 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
W20260620 18:41:21.208000 12807 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:41:21.206132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46240 (local address 127.22.37.60:45055)
0620 18:41:21.206364 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:21.206370 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:21.206628 (+ 258us) server_negotiation.cc:407] Connection header received
0620 18:41:21.206835 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:21.206842 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:21.206916 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:21.207025 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:21.207746 (+ 721us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46240: BlockingRecv error: recv got EOF from 127.0.0.1:46240 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 18:41:21.249575 12734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:21.244075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40310 (local address 127.22.37.61:45785)
0620 18:41:21.244389 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:21.244399 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:21.244941 (+ 542us) server_negotiation.cc:407] Connection header received
0620 18:41:21.245157 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:21.245164 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:21.245235 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:21.245341 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:41:21.245804 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:21.246528 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:21.247829 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:21.248269 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:21.248406 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:21.248815 (+ 409us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:41:21.248916 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:41:21.249022 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:21.249172 (+ 150us) server_negotiation.cc:299] Negotiation successful
0620 18:41:21.249254 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260620 18:41:21.257126 12807 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:41:21.255298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46242 (local address 127.22.37.60:45055)
0620 18:41:21.255492 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:21.255498 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:21.255848 (+ 350us) server_negotiation.cc:407] Connection header received
0620 18:41:21.255989 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:21.255995 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:21.256084 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:21.256185 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:21.256902 (+ 717us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46242: BlockingRecv error: recv got EOF from 127.0.0.1:46242 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 18:41:21.329299 12734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:21.324412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40326 (local address 127.22.37.61:45785)
0620 18:41:21.324589 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:21.324594 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:21.324979 (+ 385us) server_negotiation.cc:407] Connection header received
0620 18:41:21.325197 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:21.325204 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:21.325329 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:21.325487 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:41:21.325808 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:21.326541 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:21.327802 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:21.328195 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:21.328291 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:21.328692 (+ 401us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:41:21.328804 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:41:21.328901 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:21.329030 (+ 129us) server_negotiation.cc:299] Negotiation successful
0620 18:41:21.329093 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260620 18:41:21.337057 12807 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:41:21.335169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46252 (local address 127.22.37.60:45055)
0620 18:41:21.335401 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:21.335405 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:21.335799 (+ 394us) server_negotiation.cc:407] Connection header received
0620 18:41:21.335993 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:21.335998 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:21.336073 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:21.336158 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:21.336838 (+ 680us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46252: BlockingRecv error: recv got EOF from 127.0.0.1:46252 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 18:41:21.376478 12823 catalog_manager.cc:1259] Loaded cluster ID: cc9ca0ac29dc4b22abb9b8d5301b3b7a
I20260620 18:41:21.376631 12823 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b4c4770140d0412cb916a0c38d1b015c: loading cluster ID for follower catalog manager: success
I20260620 18:41:21.379348 12823 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b4c4770140d0412cb916a0c38d1b015c: acquiring CA information for follower catalog manager: success
I20260620 18:41:21.381132 12823 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b4c4770140d0412cb916a0c38d1b015c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 18:41:21.473419 12734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:21.468405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40330 (local address 127.22.37.61:45785)
0620 18:41:21.468634 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:21.468640 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:21.468865 (+ 225us) server_negotiation.cc:407] Connection header received
0620 18:41:21.469104 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:21.469108 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:21.469183 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:21.469285 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:41:21.469726 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:21.470405 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:21.471728 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:21.472152 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:21.472294 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:21.472676 (+ 382us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:41:21.472786 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:41:21.472896 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:21.473033 (+ 137us) server_negotiation.cc:299] Negotiation successful
0620 18:41:21.473114 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 18:41:21.483826 12807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:21.478724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46262 (local address 127.22.37.60:45055)
0620 18:41:21.478984 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:21.478988 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:21.479086 (+ 98us) server_negotiation.cc:407] Connection header received
0620 18:41:21.479209 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:21.479213 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:21.479356 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:21.479448 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:41:21.479797 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:21.480447 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:21.481827 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:21.482234 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:21.482322 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:21.482727 (+ 405us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:41:21.483236 (+ 509us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:41:21.483360 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:21.483437 (+ 77us) server_negotiation.cc:299] Negotiation successful
0620 18:41:21.483497 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 18:41:21.497323 12734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:21.490025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40334 (local address 127.22.37.61:45785)
0620 18:41:21.490263 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:21.490271 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:21.490983 (+ 712us) server_negotiation.cc:407] Connection header received
0620 18:41:21.491318 (+ 335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:21.491327 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:21.491430 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:21.491516 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:41:21.492278 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:21.493469 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:21.495582 (+ 2113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:21.496012 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:21.496152 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:21.496605 (+ 453us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:41:21.496714 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:41:21.496814 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:21.496992 (+ 178us) server_negotiation.cc:299] Negotiation successful
0620 18:41:21.497074 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 18:41:21.498703 12807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:21.491022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46266 (local address 127.22.37.60:45055)
0620 18:41:21.491181 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:21.491188 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:21.491704 (+ 516us) server_negotiation.cc:407] Connection header received
0620 18:41:21.491841 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:21.491847 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:21.491954 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:21.492042 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:41:21.492750 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:21.494202 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:21.496566 (+ 2364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:21.497230 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:21.497324 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:21.497959 (+ 635us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:41:21.498074 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:41:21.498169 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:21.498328 (+ 159us) server_negotiation.cc:299] Negotiation successful
0620 18:41:21.498418 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 18:41:21.499071 12977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:21.489727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36170 (local address 127.22.37.62:38759)
0620 18:41:21.490273 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:21.490279 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:21.491046 (+ 767us) server_negotiation.cc:407] Connection header received
0620 18:41:21.491379 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:21.491386 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:21.491625 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:21.491773 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:41:21.492724 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:21.493846 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:21.496220 (+ 2374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:21.496865 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:21.496977 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:21.497839 (+ 862us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:41:21.498338 (+ 499us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:41:21.498445 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:21.498641 (+ 196us) server_negotiation.cc:299] Negotiation successful
0620 18:41:21.498733 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":555,"threads_started":1}
I20260620 18:41:21.504071 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 12589
I20260620 18:41:21.505090 12651 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:41:21.616652 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 12589
W20260620 18:41:21.637630 12681 connection.cc:535] server connection from 127.0.0.1:40206 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 18:41:21.638058 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 12663
I20260620 18:41:21.639178 12725 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:41:21.744774 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 12663
I20260620 18:41:21.765292 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 12737
I20260620 18:41:21.766521 12799 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:41:21.873332 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 12737
2026-06-20T18:41:21Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4435 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980527431848-22676-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:41:21 dist-test-slave-5lgp krb5kdc[12985](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:41:21 dist-test-slave-5lgp krb5kdc[12985](info): set up 2 sockets
Jun 20 18:41:21 dist-test-slave-5lgp krb5kdc[12985](info): commencing operation
krb5kdc: starting...
W20260620 18:41:22.977883 22676 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.003s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980527431848-22676-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980527431848-22676-0/krb5kdc/test-user.keytab.
Jun 20 18:41:23 dist-test-slave-5lgp krb5kdc[12985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980883, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:41:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:41:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.37.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.37.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.37.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab.
Entry for principal kudu/127.22.37.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab.
I20260620 18:41:23.230831 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980527431848-22676-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980527431848-22676-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980527431848-22676-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.62:35829
--webserver_interface=127.22.37.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab
--principal=kudu/127.22.37.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:36973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.37.62:35829,127.22.37.61:40617,127.22.37.60:46733
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:41:23.560226 12998 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 18:41:23.560662 12998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:41:23.560792 12998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:41:23.560950 12998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:41:23.568257 12998 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:41:23.568387 12998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:41:23.568476 12998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:41:23.568553 12998 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:41:23.568641 12998 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:41:23.578578 12998 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:36973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980527431848-22676-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980527431848-22676-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.37.62:35829,127.22.37.61:40617,127.22.37.60:46733
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.62.keytab
--principal=kudu/127.22.37.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.62:35829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.37.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980527431848-22676-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:41:23.579967 12998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:41:23.581324 12998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:41:23.593007 13003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:23.593093 13004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:23.593048 13006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:41:23.594341 12998 server_base.cc:1034] running on GCE node
I20260620 18:41:23.595261 12998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:41:23.596657 12998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:41:23.597904 12998 hybrid_clock.cc:648] HybridClock initialized: now 1781980883597851 us; error 40 us; skew 500 ppm
I20260620 18:41:23.598234 12998 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:41:23 dist-test-slave-5lgp krb5kdc[12985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980883, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:41:23.601423 12998 init.cc:377] Logged in from keytab as kudu/127.22.37.62@KRBTEST.COM (short username kudu)
I20260620 18:41:23.603018 12998 webserver.cc:460] Webserver started at http://127.22.37.62:32983/ using document root <none> and password file <none>
I20260620 18:41:23.603705 12998 fs_manager.cc:359] Metadata directory not provided
I20260620 18:41:23.603839 12998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:41:23.604157 12998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:41:23.607151 12998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980527431848-22676-0/minicluster-data/master-0/data/instance:
uuid: "833cb7be5aa0462fbe0003338d84b367"
format_stamp: "Formatted at 2026-06-20 18:41:23 on dist-test-slave-5lgp"
server_key: "c9489ded3bb44c1c8da11f12dcb87f96"
server_key_iv: "8afe9421dd6d0dcd26dc9956c16ca2a7"
server_key_version: "encryptionkey@0"
I20260620 18:41:23.607947 12998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980527431848-22676-0/minicluster-data/master-0/wal/instance:
uuid: "833cb7be5aa0462fbe0003338d84b367"
format_stamp: "Formatted at 2026-06-20 18:41:23 on dist-test-slave-5lgp"
server_key: "c9489ded3bb44c1c8da11f12dcb87f96"
server_key_iv: "8afe9421dd6d0dcd26dc9956c16ca2a7"
server_key_version: "encryptionkey@0"
I20260620 18:41:23.613003 12998 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 18:41:23.616583 13013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:23.618074 12998 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:41:23.618258 12998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980527431848-22676-0/minicluster-data/master-0/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980527431848-22676-0/minicluster-data/master-0/wal
uuid: "833cb7be5aa0462fbe0003338d84b367"
format_stamp: "Formatted at 2026-06-20 18:41:23 on dist-test-slave-5lgp"
server_key: "c9489ded3bb44c1c8da11f12dcb87f96"
server_key_iv: "8afe9421dd6d0dcd26dc9956c16ca2a7"
server_key_version: "encryptionkey@0"
I20260620 18:41:23.618484 12998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980527431848-22676-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980527431848-22676-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980527431848-22676-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:41:23.630868 12998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:41:23.634850 12998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:41:23.635114 12998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:41:23.653636 12998 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.62:35829
I20260620 18:41:23.653661 13064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.62:35829 every 8 connection(s)
I20260620 18:41:23.655015 12998 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980527431848-22676-0/minicluster-data/master-0/data/info.pb
I20260620 18:41:23.655715 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 12998
I20260620 18:41:23.656003 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980527431848-22676-0/minicluster-data/master-0/wal/instance
I20260620 18:41:23.656675 22676 external_mini_cluster.cc:1428] Setting key e362b7c7119e6636a78b3538f69255bc
I20260620 18:41:23.659004 13065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:41:23.665580 13065 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.37.62" port: 35829 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.22.37.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.37.61@KRBTEST.COM" created.
Jun 20 18:41:23 dist-test-slave-5lgp krb5kdc[12985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980883, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.62@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM
I20260620 18:41:23.692938 13068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:23.669218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:35829 (local address 127.0.0.1:51884)
0620 18:41:23.670145 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:23.670196 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:23.672262 (+ 2066us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:23.673231 (+ 969us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:23.673236 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:23.673914 (+ 678us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:41:23.674252 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:23.674269 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:23.676623 (+ 2354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:23.676627 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:23.677899 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:23.677911 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:23.679402 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:23.679407 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:23.679558 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:23.680241 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:41:23.680262 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:41:23.685116 (+ 4854us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:41:23.688531 (+ 3415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:23.688547 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:23.688563 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:23.689138 (+ 575us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:23.689619 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:23.689627 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:23.689634 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:23.689865 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:23.690696 (+ 831us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:41:23.690707 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:41:23.691009 (+ 302us) client_negotiation.cc:769] Sending connection context
0620 18:41:23.691302 (+ 293us) client_negotiation.cc:240] Negotiation successful
0620 18:41:23.691981 (+ 679us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"mutex_wait_us":108,"thread_start_us":430,"threads_started":1}
I20260620 18:41:23.692984 13069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:23.670652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51884 (local address 127.22.37.62:35829)
0620 18:41:23.671277 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:23.671286 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:23.671351 (+ 65us) server_negotiation.cc:407] Connection header received
0620 18:41:23.672650 (+ 1299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:23.672661 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:23.672966 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:23.673169 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:23.674476 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:23.676454 (+ 1978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:23.678046 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:23.679040 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:23.679233 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:23.685474 (+ 6241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:23.685486 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:23.685501 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:23.685539 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:23.688377 (+ 2838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:23.689275 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:23.689282 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:23.689284 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:23.689506 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:23.690002 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:23.690005 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:23.690007 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:23.690547 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:23.690709 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:23.691437 (+ 728us) server_negotiation.cc:299] Negotiation successful
0620 18:41:23.691780 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":411,"thread_start_us":274,"threads_started":1}
I20260620 18:41:23.696403 13065 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.37.61" port: 40617 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:41:23.697651 13068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:23.697111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.61:40617 (local address 127.0.0.1:39564)
0620 18:41:23.697320 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:23.697492 (+ 172us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.61:40617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260620 18:41:23.698016 13016 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.37.61:40617: connect: Connection refused (error 111)
I20260620 18:41:23.699761 13068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:23.699289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.61:40617 (local address 127.0.0.1:39580)
0620 18:41:23.699495 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:23.699588 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.61:40617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260620 18:41:23.699977 13065 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.37.61:40617: Network error: Client connection negotiation failed: client connection to 127.22.37.61:40617: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.37.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.61.keytab.
Entry for principal kudu/127.22.37.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.61.keytab.
I20260620 18:41:23.715725 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980527431848-22676-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980527431848-22676-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980527431848-22676-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980527431848-22676-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.61:40617
--webserver_interface=127.22.37.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.61.keytab
--principal=kudu/127.22.37.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:36973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.37.62:35829,127.22.37.61:40617,127.22.37.60:46733
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:41:23.749351 13065 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.37.61" port: 40617 } attempt: 1
I20260620 18:41:23.750631 13068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:23.750079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.61:40617 (local address 127.0.0.1:39582)
0620 18:41:23.750305 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:23.750460 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.61:40617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260620 18:41:23.751705 13068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:23.751302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.61:40617 (local address 127.0.0.1:39588)
0620 18:41:23.751464 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:23.751547 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.61:40617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 18:41:23.751901 13065 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.37.61:40617: Network error: Client connection negotiation failed: client connection to 127.22.37.61:40617: connect: Connection refused (error 111)
I20260620 18:41:23.820204 13065 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.37.61" port: 40617 } attempt: 2
I20260620 18:41:23.821733 13068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:23.820992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.61:40617 (local address 127.0.0.1:39590)
0620 18:41:23.821309 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:23.821492 (+ 183us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.61:40617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260620 18:41:23.822932 13068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:23.822448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.61:40617 (local address 127.0.0.1:39592)
0620 18:41:23.822628 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:23.822748 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.61:40617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 18:41:23.823197 13065 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.37.61:40617: Network error: Client connection negotiation failed: client connection to 127.22.37.61:40617: connect: Connection refused (error 111)
I20260620 18:41:23.914480 13065 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.37.61" port: 40617 } attempt: 3
I20260620 18:41:23.915807 13068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:23.915221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.61:40617 (local address 127.0.0.1:39602)
0620 18:41:23.915508 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:23.915638 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.61:40617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260620 18:41:23.916985 13068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:23.916553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.61:40617 (local address 127.0.0.1:39612)
0620 18:41:23.916704 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:23.916789 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.61:40617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 18:41:23.917249 13065 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.37.61:40617: Network error: Client connection negotiation failed: client connection to 127.22.37.61:40617: connect: Connection refused (error 111)
W20260620 18:41:24.040042 13072 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 18:41:24.040412 13072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:41:24.040537 13072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:41:24.040683 13072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:41:24.048012 13072 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:41:24.048099 13072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:41:24.048193 13072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:41:24.048286 13072 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:41:24.048396 13072 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:41:24.058957 13072 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:36973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980527431848-22676-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980527431848-22676-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.22.37.62:35829,127.22.37.61:40617,127.22.37.60:46733
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.61.keytab
--principal=kudu/127.22.37.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.61:40617
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980527431848-22676-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.22.37.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980527431848-22676-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:41:24.060287 13072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:41:24.060523 13065 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.37.61" port: 40617 } attempt: 4
I20260620 18:41:24.061772 13068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:24.061143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.61:40617 (local address 127.0.0.1:39614)
0620 18:41:24.061504 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:24.061615 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.61:40617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174}
I20260620 18:41:24.061901 13072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 18:41:24.062835 13068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:24.062442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.61:40617 (local address 127.0.0.1:39626)
0620 18:41:24.062591 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:24.062689 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.61:40617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 18:41:24.063055 13065 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.37.61:40617: Network error: Client connection negotiation failed: client connection to 127.22.37.61:40617: connect: Connection refused (error 111)
W20260620 18:41:24.074136 13077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:24.074066 13080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:24.074083 13078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:41:24.075760 13072 server_base.cc:1034] running on GCE node
I20260620 18:41:24.076501 13072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:41:24.077957 13072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:41:24.079200 13072 hybrid_clock.cc:648] HybridClock initialized: now 1781980884079136 us; error 42 us; skew 500 ppm
I20260620 18:41:24.079545 13072 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:41:24 dist-test-slave-5lgp krb5kdc[12985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980884, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:41:24.082643 13072 init.cc:377] Logged in from keytab as kudu/127.22.37.61@KRBTEST.COM (short username kudu)
I20260620 18:41:24.084009 13072 webserver.cc:460] Webserver started at http://127.22.37.61:34065/ using document root <none> and password file <none>
I20260620 18:41:24.084700 13072 fs_manager.cc:359] Metadata directory not provided
I20260620 18:41:24.084830 13072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:41:24.085160 13072 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:41:24.088105 13072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980527431848-22676-0/minicluster-data/master-1/data/instance:
uuid: "2df3a14eb96b4cfa9cb5dd347214d863"
format_stamp: "Formatted at 2026-06-20 18:41:24 on dist-test-slave-5lgp"
server_key: "5c770630abdfb2fb311c067769848657"
server_key_iv: "9b458a5ed485596355bede675afb366e"
server_key_version: "encryptionkey@0"
I20260620 18:41:24.088915 13072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980527431848-22676-0/minicluster-data/master-1/wal/instance:
uuid: "2df3a14eb96b4cfa9cb5dd347214d863"
format_stamp: "Formatted at 2026-06-20 18:41:24 on dist-test-slave-5lgp"
server_key: "5c770630abdfb2fb311c067769848657"
server_key_iv: "9b458a5ed485596355bede675afb366e"
server_key_version: "encryptionkey@0"
I20260620 18:41:24.093930 13072 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 18:41:24.097234 13087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:24.098598 13072 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:41:24.098798 13072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980527431848-22676-0/minicluster-data/master-1/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980527431848-22676-0/minicluster-data/master-1/wal
uuid: "2df3a14eb96b4cfa9cb5dd347214d863"
format_stamp: "Formatted at 2026-06-20 18:41:24 on dist-test-slave-5lgp"
server_key: "5c770630abdfb2fb311c067769848657"
server_key_iv: "9b458a5ed485596355bede675afb366e"
server_key_version: "encryptionkey@0"
I20260620 18:41:24.099015 13072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980527431848-22676-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980527431848-22676-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980527431848-22676-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:41:24.155989 13072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:41:24.160066 13072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:41:24.160367 13072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:41:24.178773 13072 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.61:40617
I20260620 18:41:24.178793 13138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.61:40617 every 8 connection(s)
I20260620 18:41:24.180155 13072 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980527431848-22676-0/minicluster-data/master-1/data/info.pb
I20260620 18:41:24.180629 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 13072
I20260620 18:41:24.180809 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980527431848-22676-0/minicluster-data/master-1/wal/instance
I20260620 18:41:24.181422 22676 external_mini_cluster.cc:1428] Setting key 765d2c1a81f598d11b362c5d43aeac7d
I20260620 18:41:24.184158 13139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:41:24.190862 13139 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.37.62" port: 35829 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.22.37.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.37.60@KRBTEST.COM" created.
Jun 20 18:41:24 dist-test-slave-5lgp krb5kdc[12985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980884, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.61@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM
I20260620 18:41:24.214486 13069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:24.193323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51900 (local address 127.22.37.62:35829)
0620 18:41:24.193578 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:24.193586 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:24.194813 (+ 1227us) server_negotiation.cc:407] Connection header received
0620 18:41:24.196613 (+ 1800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:24.196622 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:24.196710 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:24.196828 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:24.198324 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:24.199507 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:24.201060 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:24.201777 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:24.201931 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:24.208102 (+ 6171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:24.208107 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:24.208110 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:24.208145 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:24.210861 (+ 2716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:24.211811 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:24.211815 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:24.211817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:24.211942 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:24.212421 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:24.212425 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:24.212428 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:24.212701 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:24.212808 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:24.213804 (+ 996us) server_negotiation.cc:299] Negotiation successful
0620 18:41:24.214051 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 18:41:24.215759 13142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:24.193817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:35829 (local address 127.0.0.1:51900)
0620 18:41:24.194635 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:24.194686 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:24.196360 (+ 1674us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:24.197018 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:24.197025 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:24.197626 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:41:24.198153 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:24.198169 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:24.199676 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:24.199680 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:24.200898 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:24.200910 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:24.202022 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:24.202032 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:24.202249 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:24.202996 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:41:24.203017 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:41:24.207854 (+ 4837us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:41:24.211013 (+ 3159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:24.211024 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:24.211044 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:24.211689 (+ 645us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:24.212048 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:24.212055 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:24.212062 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:24.212297 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:24.212829 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:41:24.212840 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:41:24.213332 (+ 492us) client_negotiation.cc:769] Sending connection context
0620 18:41:24.213688 (+ 356us) client_negotiation.cc:240] Negotiation successful
0620 18:41:24.214286 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":346,"threads_started":1}
I20260620 18:41:24.217554 13139 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.37.61" port: 40617 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:41:24 dist-test-slave-5lgp krb5kdc[12985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980884, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.61@KRBTEST.COM for kudu/127.22.37.61@KRBTEST.COM
I20260620 18:41:24.233783 13142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:24.218412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.61:40617 (local address 127.0.0.1:39634)
0620 18:41:24.218649 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:24.218675 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:24.218787 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:24.219552 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:24.219555 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:24.219903 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:41:24.220041 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:24.220048 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:24.222151 (+ 2103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:24.222155 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:24.223060 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:24.223072 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:24.223921 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:24.223926 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:24.224162 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:24.224748 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:41:24.224764 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:41:24.226751 (+ 1987us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:41:24.231242 (+ 4491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:24.231248 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:24.231254 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:24.231626 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:24.232093 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:24.232097 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:24.232099 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:24.232226 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:24.233016 (+ 790us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:41:24.233019 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:41:24.233130 (+ 111us) client_negotiation.cc:769] Sending connection context
0620 18:41:24.233238 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 18:41:24.233498 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62,"mutex_wait_us":24}
I20260620 18:41:24.234058 13144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:24.218414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39634 (local address 127.22.37.61:40617)
0620 18:41:24.219042 (+ 628us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:24.219052 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:24.219087 (+ 35us) server_negotiation.cc:407] Connection header received
0620 18:41:24.219144 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:24.219155 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:24.219390 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:24.219558 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:24.220225 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:24.222012 (+ 1787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:24.223226 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:24.223767 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:24.223902 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:24.226976 (+ 3074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:24.227004 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:24.227031 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:24.227093 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:24.231066 (+ 3973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:24.231785 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:24.231796 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:24.231800 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:24.231978 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:24.232342 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:24.232350 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:24.232353 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:24.232869 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:24.233093 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:24.233476 (+ 383us) server_negotiation.cc:299] Negotiation successful
0620 18:41:24.233688 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":432,"thread_start_us":283,"threads_started":1}
I20260620 18:41:24.236737 13139 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.37.60" port: 46733 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:41:24.238065 13142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:24.237596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.60:46733 (local address 127.0.0.1:48300)
0620 18:41:24.237757 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:24.237872 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.60:46733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260620 18:41:24.238371 13089 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.37.60:46733: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.37.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.60.keytab.
Entry for principal kudu/127.22.37.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.60.keytab.
I20260620 18:41:24.239876 13142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:24.239551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.60:46733 (local address 127.0.0.1:48308)
0620 18:41:24.239697 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:24.239770 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.60:46733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260620 18:41:24.240145 13139 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.37.60:46733: Network error: Client connection negotiation failed: client connection to 127.22.37.60:46733: connect: Connection refused (error 111)
I20260620 18:41:24.240687 22676 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1UXpXs/build/asan/bin/kudu
/tmp/dist-test-task1UXpXs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980527431848-22676-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980527431848-22676-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980527431848-22676-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980527431848-22676-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.60:46733
--webserver_interface=127.22.37.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.60.keytab
--principal=kudu/127.22.37.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.37.20:36973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.37.62:35829,127.22.37.61:40617,127.22.37.60:46733
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980527431848-22676-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980527431848-22676-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980527431848-22676-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:41:24.289474 13139 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.37.60" port: 46733 } attempt: 1
I20260620 18:41:24.290706 13142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:24.290260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.60:46733 (local address 127.0.0.1:48310)
0620 18:41:24.290460 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:24.290558 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.60:46733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260620 18:41:24.291703 13142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:24.291348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.60:46733 (local address 127.0.0.1:48322)
0620 18:41:24.291521 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:24.291595 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.60:46733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 18:41:24.291915 13139 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.37.60:46733: Network error: Client connection negotiation failed: client connection to 127.22.37.60:46733: connect: Connection refused (error 111)
I20260620 18:41:24.360123 13139 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.37.60" port: 46733 } attempt: 2
I20260620 18:41:24.361303 13142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:24.360812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.60:46733 (local address 127.0.0.1:48324)
0620 18:41:24.361044 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:24.361142 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.60:46733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260620 18:41:24.362339 13142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:24.361980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.60:46733 (local address 127.0.0.1:48338)
0620 18:41:24.362122 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:24.362191 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.60:46733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 18:41:24.362335 13065 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.37.61" port: 40617 } attempt: 5
W20260620 18:41:24.362767 13139 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.37.60:46733: Network error: Client connection negotiation failed: client connection to 127.22.37.60:46733: connect: Connection refused (error 111)
Jun 20 18:41:24 dist-test-slave-5lgp krb5kdc[12985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980883, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.62@KRBTEST.COM for kudu/127.22.37.61@KRBTEST.COM
I20260620 18:41:24.375813 13068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:24.362968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.61:40617 (local address 127.0.0.1:39636)
0620 18:41:24.363160 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:24.363179 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:24.363317 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:24.363680 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:24.363685 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:24.364106 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:41:24.364278 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:24.364286 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:24.365287 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:24.365293 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:24.366224 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:24.366236 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:24.367006 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:24.367012 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:24.367216 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:24.367869 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:41:24.367888 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:41:24.369954 (+ 2066us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:41:24.373581 (+ 3627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:24.373585 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:24.373588 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:24.373962 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:24.374360 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:24.374364 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:24.374366 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:24.374498 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:24.375148 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:41:24.375151 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:41:24.375277 (+ 126us) client_negotiation.cc:769] Sending connection context
0620 18:41:24.375380 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 18:41:24.375571 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260620 18:41:24.376066 13144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:24.362995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39636 (local address 127.22.37.61:40617)
0620 18:41:24.363154 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:24.363159 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:24.363233 (+ 74us) server_negotiation.cc:407] Connection header received
0620 18:41:24.363451 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:24.363456 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:24.363542 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:24.363646 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:24.364428 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:24.365150 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:24.366387 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:24.366842 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:24.366984 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:24.370160 (+ 3176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:24.370167 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:24.370171 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:24.370210 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:24.373407 (+ 3197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:24.374080 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:24.374086 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:24.374089 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:24.374243 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:24.374612 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:24.374619 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:24.374624 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:24.375026 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:24.375175 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:24.375519 (+ 344us) server_negotiation.cc:299] Negotiation successful
0620 18:41:24.375752 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 18:41:24.376998 13065 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.37.60" port: 46733 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:41:24.378185 13068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:24.377806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.60:46733 (local address 127.0.0.1:48348)
0620 18:41:24.377982 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:24.378052 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.60:46733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260620 18:41:24.379165 13068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:24.378779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.60:46733 (local address 127.0.0.1:48356)
0620 18:41:24.378948 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:24.379044 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.60:46733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260620 18:41:24.379398 13065 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.37.60:46733: Network error: Client connection negotiation failed: client connection to 127.22.37.60:46733: connect: Connection refused (error 111)
I20260620 18:41:24.430651 13065 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.37.60" port: 46733 } attempt: 1
I20260620 18:41:24.431998 13068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:24.431323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.60:46733 (local address 127.0.0.1:48362)
0620 18:41:24.431592 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:24.431735 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.60:46733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260620 18:41:24.433302 13068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:24.432836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.60:46733 (local address 127.0.0.1:48372)
0620 18:41:24.433026 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:24.433137 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.60:46733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 18:41:24.433585 13065 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.37.60:46733: Network error: Client connection negotiation failed: client connection to 127.22.37.60:46733: connect: Connection refused (error 111)
I20260620 18:41:24.454007 13139 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.37.60" port: 46733 } attempt: 3
I20260620 18:41:24.455127 13142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:24.454659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.60:46733 (local address 127.0.0.1:48384)
0620 18:41:24.454873 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:24.454966 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.60:46733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260620 18:41:24.456146 13142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:24.455742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.60:46733 (local address 127.0.0.1:48396)
0620 18:41:24.455958 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:24.456036 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.60:46733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
W20260620 18:41:24.456334 13139 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.37.60:46733: Network error: Client connection negotiation failed: client connection to 127.22.37.60:46733: connect: Connection refused (error 111)
I20260620 18:41:24.501824 13065 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.37.60" port: 46733 } attempt: 2
I20260620 18:41:24.503010 13068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:24.502559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.60:46733 (local address 127.0.0.1:48410)
0620 18:41:24.502752 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:24.502861 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.60:46733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260620 18:41:24.504033 13068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:24.503694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.60:46733 (local address 127.0.0.1:48426)
0620 18:41:24.503848 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:24.503918 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.60:46733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 18:41:24.504261 13065 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.37.60:46733: Network error: Client connection negotiation failed: client connection to 127.22.37.60:46733: connect: Connection refused (error 111)
W20260620 18:41:24.569272 13146 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 18:41:24.569707 13146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:41:24.569842 13146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:41:24.569981 13146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:41:24.577463 13146 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:41:24.577585 13146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:41:24.577704 13146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:41:24.577796 13146 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:41:24.577884 13146 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:41:24.587900 13146 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.37.20:36973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980527431848-22676-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980527431848-22676-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.22.37.62:35829,127.22.37.61:40617,127.22.37.60:46733
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980527431848-22676-0/krb5kdc/kudu_127.22.37.60.keytab
--principal=kudu/127.22.37.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.37.60:46733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980527431848-22676-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.22.37.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980527431848-22676-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:23:24 UTC on ccycloud.quasar-worxty.root.comops.site
build id 4456964
ASAN enabled
I20260620 18:41:24.589501 13146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:41:24.590801 13146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 18:41:24.599538 13139 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.37.60" port: 46733 } attempt: 4
I20260620 18:41:24.600633 13142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:24.600185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.60:46733 (local address 127.0.0.1:48438)
0620 18:41:24.600380 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:24.600467 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.60:46733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260620 18:41:24.601814 13142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:24.601450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.60:46733 (local address 127.0.0.1:48444)
0620 18:41:24.601647 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:24.601710 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.60:46733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260620 18:41:24.602129 13139 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.37.60:46733: Network error: Client connection negotiation failed: client connection to 127.22.37.60:46733: connect: Connection refused (error 111)
W20260620 18:41:24.602630 13151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:24.602573 13152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:24.604031 13154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:41:24.604260 13146 server_base.cc:1034] running on GCE node
I20260620 18:41:24.605098 13146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:41:24.606582 13146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:41:24.607817 13146 hybrid_clock.cc:648] HybridClock initialized: now 1781980884607745 us; error 56 us; skew 500 ppm
I20260620 18:41:24.608142 13146 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:41:24.610513 13065 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.37.60" port: 46733 } attempt: 3
Jun 20 18:41:24 dist-test-slave-5lgp krb5kdc[12985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980884, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:41:24.611382 13146 init.cc:377] Logged in from keytab as kudu/127.22.37.60@KRBTEST.COM (short username kudu)
I20260620 18:41:24.611642 13068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:24.611123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.60:46733 (local address 127.0.0.1:48446)
0620 18:41:24.611349 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:24.611429 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.60:46733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 18:41:24.612882 13068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:24.612400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.60:46733 (local address 127.0.0.1:48456)
0620 18:41:24.612663 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:24.612739 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.37.60:46733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260620 18:41:24.613107 13146 webserver.cc:460] Webserver started at http://127.22.37.60:38695/ using document root <none> and password file <none>
W20260620 18:41:24.613152 13065 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.37.60:46733: Network error: Client connection negotiation failed: client connection to 127.22.37.60:46733: connect: Connection refused (error 111)
I20260620 18:41:24.614029 13146 fs_manager.cc:359] Metadata directory not provided
I20260620 18:41:24.614151 13146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:41:24.614501 13146 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:41:24.617512 13146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980527431848-22676-0/minicluster-data/master-2/data/instance:
uuid: "31d7504f52674d98964749fdda4e8dde"
format_stamp: "Formatted at 2026-06-20 18:41:24 on dist-test-slave-5lgp"
server_key: "ee24af3850be1d339f6cdd2516f8664c"
server_key_iv: "d179ab4ed5fbe78e8b908468635aab85"
server_key_version: "encryptionkey@0"
I20260620 18:41:24.618294 13146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980527431848-22676-0/minicluster-data/master-2/wal/instance:
uuid: "31d7504f52674d98964749fdda4e8dde"
format_stamp: "Formatted at 2026-06-20 18:41:24 on dist-test-slave-5lgp"
server_key: "ee24af3850be1d339f6cdd2516f8664c"
server_key_iv: "d179ab4ed5fbe78e8b908468635aab85"
server_key_version: "encryptionkey@0"
I20260620 18:41:24.623283 13146 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:41:24.626684 13161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:24.627990 13146 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:41:24.628172 13146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980527431848-22676-0/minicluster-data/master-2/data,/tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980527431848-22676-0/minicluster-data/master-2/wal
uuid: "31d7504f52674d98964749fdda4e8dde"
format_stamp: "Formatted at 2026-06-20 18:41:24 on dist-test-slave-5lgp"
server_key: "ee24af3850be1d339f6cdd2516f8664c"
server_key_iv: "d179ab4ed5fbe78e8b908468635aab85"
server_key_version: "encryptionkey@0"
I20260620 18:41:24.628387 13146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980527431848-22676-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980527431848-22676-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980527431848-22676-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:41:24.643514 13146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:41:24.647866 13146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:41:24.648149 13146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:41:24.666960 13146 rpc_server.cc:307] RPC server started. Bound to: 127.22.37.60:46733
I20260620 18:41:24.666980 13212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.37.60:46733 every 8 connection(s)
I20260620 18:41:24.668315 13146 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980527431848-22676-0/minicluster-data/master-2/data/info.pb
I20260620 18:41:24.672040 13213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:41:24.675348 22676 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1UXpXs/build/asan/bin/kudu as pid 13146
I20260620 18:41:24.675554 22676 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1UXpXs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980527431848-22676-0/minicluster-data/master-2/wal/instance
I20260620 18:41:24.676005 22676 external_mini_cluster.cc:1428] Setting key c40e85127a943719b546f70f3cd24c66
I20260620 18:41:24.679090 13213 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.37.62" port: 35829 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:41:24 dist-test-slave-5lgp krb5kdc[12985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980883, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.37.60@KRBTEST.COM
Jun 20 18:41:24 dist-test-slave-5lgp krb5kdc[12985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980884, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.60@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM
I20260620 18:41:24.698696 13069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:24.682907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51906 (local address 127.22.37.62:35829)
0620 18:41:24.683127 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:24.683133 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:24.683676 (+ 543us) server_negotiation.cc:407] Connection header received
0620 18:41:24.683841 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:24.683847 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:24.683943 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:24.684052 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:24.685315 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:24.686112 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:24.687814 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:24.688233 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:24.688324 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:24.692134 (+ 3810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:24.692141 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:24.692146 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:24.692188 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:24.695478 (+ 3290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:24.696313 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:24.696319 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:24.696322 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:24.696501 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:24.696917 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:24.696922 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:24.696925 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:24.697245 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:24.697445 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:24.698101 (+ 656us) server_negotiation.cc:299] Negotiation successful
0620 18:41:24.698328 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 18:41:24.699681 13217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:24.682798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.62:35829 (local address 127.0.0.1:51906)
0620 18:41:24.683505 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:24.683565 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:24.683721 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:24.684102 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:24.684106 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:24.684873 (+ 767us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:41:24.685169 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:24.685181 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:24.686269 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:24.686273 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:24.687635 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:24.687646 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:24.688471 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:24.688480 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:24.688714 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:24.689321 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:41:24.689361 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:41:24.691951 (+ 2590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:41:24.695610 (+ 3659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:24.695621 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:24.695633 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:24.696192 (+ 559us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:24.696608 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:24.696614 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:24.696617 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:24.696807 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:24.697414 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:41:24.697424 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:41:24.697670 (+ 246us) client_negotiation.cc:769] Sending connection context
0620 18:41:24.697987 (+ 317us) client_negotiation.cc:240] Negotiation successful
0620 18:41:24.698715 (+ 728us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"mutex_wait_us":274,"thread_start_us":336,"threads_started":1}
I20260620 18:41:24.699699 13216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:24.677804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48464 (local address 127.22.37.60:46733)
0620 18:41:24.678594 (+ 790us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:24.678606 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:24.678643 (+ 37us) server_negotiation.cc:407] Connection header received
0620 18:41:24.680118 (+ 1475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:24.680138 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:24.680561 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:24.680919 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:24.682729 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:24.684485 (+ 1756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:24.686030 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:24.686813 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:24.686939 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:24.691732 (+ 4793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:24.691740 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:24.691750 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:24.691777 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:24.695321 (+ 3544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:24.696126 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:24.696135 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:24.696137 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:24.696338 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:24.696753 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:24.696756 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:24.696758 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:24.697456 (+ 698us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:24.697695 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:24.698101 (+ 406us) server_negotiation.cc:299] Negotiation successful
0620 18:41:24.698463 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":588,"thread_start_us":355,"threads_started":1}
I20260620 18:41:24.701718 13213 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.37.61" port: 40617 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:41:24 dist-test-slave-5lgp krb5kdc[12985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980884, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.60@KRBTEST.COM for kudu/127.22.37.61@KRBTEST.COM
Jun 20 18:41:24 dist-test-slave-5lgp krb5kdc[12985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980883, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.37.62@KRBTEST.COM
I20260620 18:41:24.716317 13217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:24.702504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.61:40617 (local address 127.0.0.1:39644)
0620 18:41:24.702699 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:24.702718 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:24.702842 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:24.703228 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:24.703232 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:24.703699 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:41:24.703886 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:24.703894 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:24.705042 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:24.705045 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:24.706263 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:24.706277 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:24.706938 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:24.706944 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:24.707153 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:24.707732 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:41:24.707749 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:41:24.710071 (+ 2322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:41:24.713782 (+ 3711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:24.713789 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:24.713794 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:24.714172 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:24.714577 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:24.714580 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:24.714582 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:24.714710 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:24.715340 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:41:24.715343 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:41:24.715569 (+ 226us) client_negotiation.cc:769] Sending connection context
0620 18:41:24.715745 (+ 176us) client_negotiation.cc:240] Negotiation successful
0620 18:41:24.716033 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 18:41:24.716526 13144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:24.702557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39644 (local address 127.22.37.61:40617)
0620 18:41:24.702816 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:24.702822 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:24.702841 (+ 19us) server_negotiation.cc:407] Connection header received
0620 18:41:24.702965 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:24.702971 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:24.703079 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:24.703176 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:24.704085 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:24.704886 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:24.706412 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:24.706798 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:24.706901 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:24.710279 (+ 3378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:24.710286 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:24.710291 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:24.710339 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:24.713639 (+ 3300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:24.714292 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:24.714299 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:24.714302 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:24.714465 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:24.714820 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:24.714826 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:24.714830 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:24.715227 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:24.715387 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:24.715823 (+ 436us) server_negotiation.cc:299] Negotiation successful
0620 18:41:24.716002 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260620 18:41:24.717635 13213 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.37.60" port: 46733 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:41:24.718555 13069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:24.703079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51920 (local address 127.22.37.62:35829)
0620 18:41:24.703244 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:24.703252 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:24.703268 (+ 16us) server_negotiation.cc:407] Connection header received
0620 18:41:24.703321 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:24.703327 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:24.703393 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:24.703497 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:24.704682 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:24.705602 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:24.707067 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:24.707469 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:24.707603 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:24.711969 (+ 4366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:24.711994 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:24.712001 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:24.712048 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:24.715293 (+ 3245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:24.716116 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:24.716123 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:24.716128 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:24.716270 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:24.716740 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:24.716745 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:24.716748 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:24.717056 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:24.717338 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:24.717652 (+ 314us) server_negotiation.cc:299] Negotiation successful
0620 18:41:24.717872 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
Jun 20 18:41:24 dist-test-slave-5lgp krb5kdc[12985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980884, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.60@KRBTEST.COM for kudu/127.22.37.60@KRBTEST.COM
Jun 20 18:41:24 dist-test-slave-5lgp krb5kdc[12985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980883, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.37.61@KRBTEST.COM
I20260620 18:41:24.732445 13216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:24.718571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48478 (local address 127.22.37.60:46733)
0620 18:41:24.718739 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:24.718744 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:24.718937 (+ 193us) server_negotiation.cc:407] Connection header received
0620 18:41:24.719084 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:24.719089 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:24.719165 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:24.719274 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:24.719998 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:24.721042 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:24.722417 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:24.722857 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:24.723047 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:24.726336 (+ 3289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:24.726342 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:24.726349 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:24.726383 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:24.729485 (+ 3102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:24.730291 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:24.730295 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:24.730297 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:24.730461 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:24.730802 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:24.730807 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:24.730813 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:24.731172 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:24.731325 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:24.731805 (+ 480us) server_negotiation.cc:299] Negotiation successful
0620 18:41:24.732067 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":59}
I20260620 18:41:24.732735 13217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:24.718561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.60:46733 (local address 127.0.0.1:48478)
0620 18:41:24.718845 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:24.718864 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:24.718977 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:24.719299 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:24.719303 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:24.719733 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:41:24.719860 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:24.719868 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:24.721171 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:24.721175 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:24.722283 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:24.722297 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:24.722998 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:24.723005 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:24.723231 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:24.723847 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:41:24.723864 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:41:24.726145 (+ 2281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:41:24.729613 (+ 3468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:24.729619 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:24.729625 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:24.730172 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:24.730570 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:24.730573 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:24.730575 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:24.730697 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:24.731331 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:41:24.731339 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:41:24.731480 (+ 141us) client_negotiation.cc:769] Sending connection context
0620 18:41:24.731702 (+ 222us) client_negotiation.cc:240] Negotiation successful
0620 18:41:24.732232 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"mutex_wait_us":165}
I20260620 18:41:24.736639 13144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:24.720316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39658 (local address 127.22.37.61:40617)
0620 18:41:24.720474 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:24.720478 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:24.720493 (+ 15us) server_negotiation.cc:407] Connection header received
0620 18:41:24.720630 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:24.720633 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:24.720684 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:24.720758 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:24.722171 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:24.723152 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:24.724485 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:24.725114 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:24.725209 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:24.730464 (+ 5255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:24.730471 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:24.730475 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:24.730517 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:24.733916 (+ 3399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:24.734686 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:24.734692 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:24.734696 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:24.734833 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:24.735206 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:24.735212 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:24.735215 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:24.735485 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:24.735773 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:24.736062 (+ 289us) server_negotiation.cc:299] Negotiation successful
0620 18:41:24.736290 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 18:41:24.737983 13213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31d7504f52674d98964749fdda4e8dde: Bootstrap starting.
I20260620 18:41:24.741652 13213 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 31d7504f52674d98964749fdda4e8dde: Neither blocks nor log segments found. Creating new log.
I20260620 18:41:24.741698 22676 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:41:24.742995 13213 log.cc:824] T 00000000000000000000000000000000 P 31d7504f52674d98964749fdda4e8dde: Log is configured to *not* fsync() on all Append() calls
I20260620 18:41:24.745839 13213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31d7504f52674d98964749fdda4e8dde: No bootstrap required, opened a new log
I20260620 18:41:24.744691 13040 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "833cb7be5aa0462fbe0003338d84b367"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51920
I20260620 18:41:24.752884 13213 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31d7504f52674d98964749fdda4e8dde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833cb7be5aa0462fbe0003338d84b367" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 35829 } } peers { permanent_uuid: "2df3a14eb96b4cfa9cb5dd347214d863" member_type: VOTER last_known_addr { host: "127.22.37.61" port: 40617 } } peers { permanent_uuid: "31d7504f52674d98964749fdda4e8dde" member_type: VOTER last_known_addr { host: "127.22.37.60" port: 46733 } }
I20260620 18:41:24.753233 13213 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31d7504f52674d98964749fdda4e8dde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31d7504f52674d98964749fdda4e8dde, State: Initialized, Role: FOLLOWER
I20260620 18:41:24.753700 13213 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31d7504f52674d98964749fdda4e8dde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833cb7be5aa0462fbe0003338d84b367" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 35829 } } peers { permanent_uuid: "2df3a14eb96b4cfa9cb5dd347214d863" member_type: VOTER last_known_addr { host: "127.22.37.61" port: 40617 } } peers { permanent_uuid: "31d7504f52674d98964749fdda4e8dde" member_type: VOTER last_known_addr { host: "127.22.37.60" port: 46733 } }
I20260620 18:41:24.754631 13219 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31d7504f52674d98964749fdda4e8dde [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833cb7be5aa0462fbe0003338d84b367" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 35829 } } peers { permanent_uuid: "2df3a14eb96b4cfa9cb5dd347214d863" member_type: VOTER last_known_addr { host: "127.22.37.61" port: 40617 } } peers { permanent_uuid: "31d7504f52674d98964749fdda4e8dde" member_type: VOTER last_known_addr { host: "127.22.37.60" port: 46733 } } }
I20260620 18:41:24.755077 13219 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31d7504f52674d98964749fdda4e8dde [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:41:24.755640 13213 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31d7504f52674d98964749fdda4e8dde [sys.catalog]: configured and running, proceeding with master startup.
W20260620 18:41:24.764968 13230 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 31d7504f52674d98964749fdda4e8dde: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 18:41:24.765089 13230 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 18:41:24.762756 13146 master_runner.cc:186] Error getting master registration for 127.22.37.62:35829: OK, instance_id { permanent_uuid: "833cb7be5aa0462fbe0003338d84b367" instance_seqno: 1781980883648359 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 18:41:24.770727 13146 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 18:41:24.790381 13065 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.37.60" port: 46733 } attempt: 4
Jun 20 18:41:24 dist-test-slave-5lgp krb5kdc[12985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980883, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.62@KRBTEST.COM for kudu/127.22.37.60@KRBTEST.COM
I20260620 18:41:24.807612 13068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:24.791141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.60:46733 (local address 127.0.0.1:48488)
0620 18:41:24.791498 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:24.791519 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:24.791664 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:24.791986 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:24.791992 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:24.792507 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:41:24.792722 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:24.792736 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:24.794226 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:24.794232 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:24.795737 (+ 1505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:24.795753 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:24.796772 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:24.796779 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:24.796990 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:24.797709 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:41:24.797736 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:41:24.800527 (+ 2791us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:41:24.804720 (+ 4193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:24.804725 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:24.804730 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:24.805251 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:24.805674 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:24.805679 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:24.805683 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:24.805855 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:24.806586 (+ 731us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:41:24.806591 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:41:24.806755 (+ 164us) client_negotiation.cc:769] Sending connection context
0620 18:41:24.806897 (+ 142us) client_negotiation.cc:240] Negotiation successful
0620 18:41:24.807200 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213}
I20260620 18:41:24.807632 13216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:24.791254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48488 (local address 127.22.37.60:46733)
0620 18:41:24.791465 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:24.791469 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:24.791583 (+ 114us) server_negotiation.cc:407] Connection header received
0620 18:41:24.791782 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:24.791787 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:24.791846 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:24.791930 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:24.792875 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:24.794070 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:24.795905 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:24.796606 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:24.796732 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:24.800715 (+ 3983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:24.800722 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:24.800726 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:24.800765 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:24.804573 (+ 3808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:24.805402 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:24.805408 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:24.805412 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:24.805550 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:24.805965 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:24.805970 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:24.805973 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:24.806464 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:24.806600 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:24.807008 (+ 408us) server_negotiation.cc:299] Negotiation successful
0620 18:41:24.807245 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 18:41:24.813470 13065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 833cb7be5aa0462fbe0003338d84b367: Bootstrap starting.
I20260620 18:41:24.816587 13065 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 833cb7be5aa0462fbe0003338d84b367: Neither blocks nor log segments found. Creating new log.
I20260620 18:41:24.817693 13065 log.cc:824] T 00000000000000000000000000000000 P 833cb7be5aa0462fbe0003338d84b367: Log is configured to *not* fsync() on all Append() calls
I20260620 18:41:24.820286 13065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 833cb7be5aa0462fbe0003338d84b367: No bootstrap required, opened a new log
I20260620 18:41:24.825474 13065 raft_consensus.cc:348] T 00000000000000000000000000000000 P 833cb7be5aa0462fbe0003338d84b367 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833cb7be5aa0462fbe0003338d84b367" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 35829 } } peers { permanent_uuid: "2df3a14eb96b4cfa9cb5dd347214d863" member_type: VOTER last_known_addr { host: "127.22.37.61" port: 40617 } } peers { permanent_uuid: "31d7504f52674d98964749fdda4e8dde" member_type: VOTER last_known_addr { host: "127.22.37.60" port: 46733 } }
I20260620 18:41:24.825822 13065 raft_consensus.cc:729] T 00000000000000000000000000000000 P 833cb7be5aa0462fbe0003338d84b367 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833cb7be5aa0462fbe0003338d84b367, State: Initialized, Role: FOLLOWER
I20260620 18:41:24.826259 13065 consensus_queue.cc:260] T 00000000000000000000000000000000 P 833cb7be5aa0462fbe0003338d84b367 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833cb7be5aa0462fbe0003338d84b367" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 35829 } } peers { permanent_uuid: "2df3a14eb96b4cfa9cb5dd347214d863" member_type: VOTER last_known_addr { host: "127.22.37.61" port: 40617 } } peers { permanent_uuid: "31d7504f52674d98964749fdda4e8dde" member_type: VOTER last_known_addr { host: "127.22.37.60" port: 46733 } }
I20260620 18:41:24.827175 13233 sys_catalog.cc:455] T 00000000000000000000000000000000 P 833cb7be5aa0462fbe0003338d84b367 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833cb7be5aa0462fbe0003338d84b367" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 35829 } } peers { permanent_uuid: "2df3a14eb96b4cfa9cb5dd347214d863" member_type: VOTER last_known_addr { host: "127.22.37.61" port: 40617 } } peers { permanent_uuid: "31d7504f52674d98964749fdda4e8dde" member_type: VOTER last_known_addr { host: "127.22.37.60" port: 46733 } } }
I20260620 18:41:24.827534 13233 sys_catalog.cc:458] T 00000000000000000000000000000000 P 833cb7be5aa0462fbe0003338d84b367 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:41:24.828289 13065 sys_catalog.cc:564] T 00000000000000000000000000000000 P 833cb7be5aa0462fbe0003338d84b367 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:41:24.828434 13040 raft_consensus.cc:482] T 00000000000000000000000000000000 P 833cb7be5aa0462fbe0003338d84b367 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 18:41:24.828658 13040 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 833cb7be5aa0462fbe0003338d84b367 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:41:24.832515 13040 raft_consensus.cc:504] T 00000000000000000000000000000000 P 833cb7be5aa0462fbe0003338d84b367 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833cb7be5aa0462fbe0003338d84b367" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 35829 } } peers { permanent_uuid: "2df3a14eb96b4cfa9cb5dd347214d863" member_type: VOTER last_known_addr { host: "127.22.37.61" port: 40617 } } peers { permanent_uuid: "31d7504f52674d98964749fdda4e8dde" member_type: VOTER last_known_addr { host: "127.22.37.60" port: 46733 } }
I20260620 18:41:24.835817 13188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "833cb7be5aa0462fbe0003338d84b367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "31d7504f52674d98964749fdda4e8dde"
I20260620 18:41:24.836325 13188 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31d7504f52674d98964749fdda4e8dde [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:41:24.839762 13188 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 31d7504f52674d98964749fdda4e8dde [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 833cb7be5aa0462fbe0003338d84b367 in term 1.
I20260620 18:41:24.840492 13015 leader_election.cc:302] T 00000000000000000000000000000000 P 833cb7be5aa0462fbe0003338d84b367 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31d7504f52674d98964749fdda4e8dde, 833cb7be5aa0462fbe0003338d84b367; no voters:
I20260620 18:41:24.840997 13233 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 833cb7be5aa0462fbe0003338d84b367 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:41:24.846093 13233 raft_consensus.cc:686] T 00000000000000000000000000000000 P 833cb7be5aa0462fbe0003338d84b367 [term 1 LEADER]: Becoming Leader. State: Replica: 833cb7be5aa0462fbe0003338d84b367, State: Running, Role: LEADER
I20260620 18:41:24.846787 13233 consensus_queue.cc:237] T 00000000000000000000000000000000 P 833cb7be5aa0462fbe0003338d84b367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833cb7be5aa0462fbe0003338d84b367" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 35829 } } peers { permanent_uuid: "2df3a14eb96b4cfa9cb5dd347214d863" member_type: VOTER last_known_addr { host: "127.22.37.61" port: 40617 } } peers { permanent_uuid: "31d7504f52674d98964749fdda4e8dde" member_type: VOTER last_known_addr { host: "127.22.37.60" port: 46733 } }
I20260620 18:41:24.849530 13040 leader_election.cc:288] T 00000000000000000000000000000000 P 833cb7be5aa0462fbe0003338d84b367 [CANDIDATE]: Term 1 election: Requested vote from peers 2df3a14eb96b4cfa9cb5dd347214d863 (127.22.37.61:40617), 31d7504f52674d98964749fdda4e8dde (127.22.37.60:46733)
I20260620 18:41:24.838801 12998 master_runner.cc:186] Error getting master registration for 127.22.37.61:40617: OK, instance_id { permanent_uuid: "2df3a14eb96b4cfa9cb5dd347214d863" instance_seqno: 1781980884173564 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 18:41:24.857195 13246 sys_catalog.cc:455] T 00000000000000000000000000000000 P 833cb7be5aa0462fbe0003338d84b367 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 833cb7be5aa0462fbe0003338d84b367. Latest consensus state: current_term: 1 leader_uuid: "833cb7be5aa0462fbe0003338d84b367" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833cb7be5aa0462fbe0003338d84b367" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 35829 } } peers { permanent_uuid: "2df3a14eb96b4cfa9cb5dd347214d863" member_type: VOTER last_known_addr { host: "127.22.37.61" port: 40617 } } peers { permanent_uuid: "31d7504f52674d98964749fdda4e8dde" member_type: VOTER last_known_addr { host: "127.22.37.60" port: 46733 } } }
I20260620 18:41:24.857769 13246 sys_catalog.cc:458] T 00000000000000000000000000000000 P 833cb7be5aa0462fbe0003338d84b367 [sys.catalog]: This master's current role is: LEADER
I20260620 18:41:24.855846 12998 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260620 18:41:24.869741 13244 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 833cb7be5aa0462fbe0003338d84b367: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 18:41:24.869905 13244 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 18:41:24.870553 13249 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:41:24.872718 13249 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260620 18:41:24.887867 13019 tablet.cc:2369] T 00000000000000000000000000000000 P 833cb7be5aa0462fbe0003338d84b367: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:41:24.897339 13188 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 31d7504f52674d98964749fdda4e8dde [term 1 FOLLOWER]: Refusing update from remote peer 833cb7be5aa0462fbe0003338d84b367: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:41:24.898015 13069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:24.872752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51922 (local address 127.22.37.62:35829)
0620 18:41:24.872938 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:24.872945 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:24.874448 (+ 1503us) server_negotiation.cc:407] Connection header received
0620 18:41:24.874634 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:24.874640 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:24.874718 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:24.874826 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:24.876894 (+ 2068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:24.878045 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:24.881256 (+ 3211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:24.882012 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:24.882113 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:24.886991 (+ 4878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:24.887001 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:24.887006 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:24.887051 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:24.892100 (+ 5049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:24.893644 (+ 1544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:24.893652 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:24.893681 (+ 29us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:24.893980 (+ 299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:24.894658 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:24.894691 (+ 33us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:24.894695 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:24.895355 (+ 660us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:24.895788 (+ 433us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:24.896751 (+ 963us) server_negotiation.cc:299] Negotiation successful
0620 18:41:24.897179 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 18:41:24.898929 13216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:24.873026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48498 (local address 127.22.37.60:46733)
0620 18:41:24.873201 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:24.873207 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:24.877326 (+ 4119us) server_negotiation.cc:407] Connection header received
0620 18:41:24.877612 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:24.877619 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:24.877715 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:24.877787 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:24.880118 (+ 2331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:24.881236 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:24.883033 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:24.883795 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:24.883907 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:24.888854 (+ 4947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:24.888862 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:24.888867 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:24.888920 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:24.892686 (+ 3766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:24.893597 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:24.893604 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:24.893610 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:24.893774 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:24.894247 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:24.894252 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:24.894256 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:24.894611 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:24.894854 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:24.898295 (+ 3441us) server_negotiation.cc:299] Negotiation successful
0620 18:41:24.898548 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260620 18:41:24.900221 13167 tablet.cc:2369] T 00000000000000000000000000000000 P 31d7504f52674d98964749fdda4e8dde: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:41:24.901077 13246 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 833cb7be5aa0462fbe0003338d84b367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31d7504f52674d98964749fdda4e8dde" member_type: VOTER last_known_addr { host: "127.22.37.60" port: 46733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:41:24.901661 13139 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.37.60" port: 46733 } attempt: 5
I20260620 18:41:24.908439 13144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:24.873878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39674 (local address 127.22.37.61:40617)
0620 18:41:24.874072 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:24.874078 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:24.885627 (+ 11549us) server_negotiation.cc:407] Connection header received
0620 18:41:24.885783 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:24.885790 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:24.885885 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:24.885992 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:24.887781 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:24.888858 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:24.890773 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:24.891454 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:24.891552 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:24.899914 (+ 8362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:24.899921 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:24.899927 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:24.899972 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:24.904585 (+ 4613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:24.905488 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:24.905493 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:24.905496 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:24.905640 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:24.906109 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:24.906113 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:24.906116 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:24.906486 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:24.906730 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:24.907821 (+ 1091us) server_negotiation.cc:299] Negotiation successful
0620 18:41:24.908058 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
Jun 20 18:41:24 dist-test-slave-5lgp krb5kdc[12985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980884, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.37.61@KRBTEST.COM for kudu/127.22.37.60@KRBTEST.COM
I20260620 18:41:24.924450 13219 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31d7504f52674d98964749fdda4e8dde [sys.catalog]: SysCatalogTable state changed. Reason: New leader 833cb7be5aa0462fbe0003338d84b367. Latest consensus state: current_term: 1 leader_uuid: "833cb7be5aa0462fbe0003338d84b367" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833cb7be5aa0462fbe0003338d84b367" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 35829 } } peers { permanent_uuid: "2df3a14eb96b4cfa9cb5dd347214d863" member_type: VOTER last_known_addr { host: "127.22.37.61" port: 40617 } } peers { permanent_uuid: "31d7504f52674d98964749fdda4e8dde" member_type: VOTER last_known_addr { host: "127.22.37.60" port: 46733 } } }
I20260620 18:41:24.924748 13219 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31d7504f52674d98964749fdda4e8dde [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:41:24.929826 13142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:24.902204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.37.60:46733 (local address 127.0.0.1:48504)
0620 18:41:24.902676 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:24.902975 (+ 299us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:24.903091 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:24.908682 (+ 5591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:24.908688 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:24.909064 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:41:24.909209 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:24.909217 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:24.912324 (+ 3107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:24.912331 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:24.913784 (+ 1453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:24.913797 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:24.914791 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:24.914797 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:24.915045 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:24.916245 (+ 1200us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:41:24.916266 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:41:24.920269 (+ 4003us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:41:24.926901 (+ 6632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:24.926907 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:24.926912 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:24.927451 (+ 539us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:24.927840 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:24.927845 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:24.927849 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:24.928032 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:24.928693 (+ 661us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:41:24.928698 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:41:24.928889 (+ 191us) client_negotiation.cc:769] Sending connection context
0620 18:41:24.929056 (+ 167us) client_negotiation.cc:240] Negotiation successful
0620 18:41:24.929418 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334}
I20260620 18:41:24.932364 13216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:24.902341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48504 (local address 127.22.37.60:46733)
0620 18:41:24.907569 (+ 5228us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:24.907575 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:24.907699 (+ 124us) server_negotiation.cc:407] Connection header received
0620 18:41:24.907852 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:24.907954 (+ 102us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:24.908251 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:24.909643 (+ 1392us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:24.910337 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:24.912185 (+ 1848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:24.913934 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:24.914655 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:24.914776 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:24.920527 (+ 5751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:24.920533 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:24.920537 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:24.920575 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:24.926766 (+ 6191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:24.927573 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:24.927579 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:24.927582 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:24.927729 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:24.928173 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:24.928179 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:24.928184 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:24.928545 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:24.928697 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:24.929175 (+ 478us) server_negotiation.cc:299] Negotiation successful
0620 18:41:24.929597 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4506}
I20260620 18:41:24.939726 13139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2df3a14eb96b4cfa9cb5dd347214d863: Bootstrap starting.
I20260620 18:41:24.943960 13233 sys_catalog.cc:455] T 00000000000000000000000000000000 P 833cb7be5aa0462fbe0003338d84b367 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "833cb7be5aa0462fbe0003338d84b367" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833cb7be5aa0462fbe0003338d84b367" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 35829 } } peers { permanent_uuid: "2df3a14eb96b4cfa9cb5dd347214d863" member_type: VOTER last_known_addr { host: "127.22.37.61" port: 40617 } } peers { permanent_uuid: "31d7504f52674d98964749fdda4e8dde" member_type: VOTER last_known_addr { host: "127.22.37.60" port: 46733 } } }
I20260620 18:41:24.944274 13233 sys_catalog.cc:458] T 00000000000000000000000000000000 P 833cb7be5aa0462fbe0003338d84b367 [sys.catalog]: This master's current role is: LEADER
I20260620 18:41:24.944644 13257 mvcc.cc:205] Tried to move back new op lower bound from 7298993704525373440 to 7298993704338649088. Current Snapshot: MvccSnapshot[applied={T|T < 7298993704525373440}]
I20260620 18:41:24.944864 13139 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2df3a14eb96b4cfa9cb5dd347214d863: Neither blocks nor log segments found. Creating new log.
I20260620 18:41:24.946777 13139 log.cc:824] T 00000000000000000000000000000000 P 2df3a14eb96b4cfa9cb5dd347214d863: Log is configured to *not* fsync() on all Append() calls
I20260620 18:41:24.949985 13219 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31d7504f52674d98964749fdda4e8dde [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "833cb7be5aa0462fbe0003338d84b367" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833cb7be5aa0462fbe0003338d84b367" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 35829 } } peers { permanent_uuid: "2df3a14eb96b4cfa9cb5dd347214d863" member_type: VOTER last_known_addr { host: "127.22.37.61" port: 40617 } } peers { permanent_uuid: "31d7504f52674d98964749fdda4e8dde" member_type: VOTER last_known_addr { host: "127.22.37.60" port: 46733 } } }
I20260620 18:41:24.950284 13219 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31d7504f52674d98964749fdda4e8dde [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:41:24.952188 13249 catalog_manager.cc:1347] Generated new cluster ID: ca805dac8b8a49af97960e75931e9183
I20260620 18:41:24.952297 13249 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:41:24.961309 13139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2df3a14eb96b4cfa9cb5dd347214d863: No bootstrap required, opened a new log
I20260620 18:41:24.974983 13139 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2df3a14eb96b4cfa9cb5dd347214d863 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833cb7be5aa0462fbe0003338d84b367" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 35829 } } peers { permanent_uuid: "2df3a14eb96b4cfa9cb5dd347214d863" member_type: VOTER last_known_addr { host: "127.22.37.61" port: 40617 } } peers { permanent_uuid: "31d7504f52674d98964749fdda4e8dde" member_type: VOTER last_known_addr { host: "127.22.37.60" port: 46733 } }
I20260620 18:41:24.975379 13139 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2df3a14eb96b4cfa9cb5dd347214d863 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2df3a14eb96b4cfa9cb5dd347214d863, State: Initialized, Role: FOLLOWER
I20260620 18:41:24.975880 13139 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2df3a14eb96b4cfa9cb5dd347214d863 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833cb7be5aa0462fbe0003338d84b367" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 35829 } } peers { permanent_uuid: "2df3a14eb96b4cfa9cb5dd347214d863" member_type: VOTER last_known_addr { host: "127.22.37.61" port: 40617 } } peers { permanent_uuid: "31d7504f52674d98964749fdda4e8dde" member_type: VOTER last_known_addr { host: "127.22.37.60" port: 46733 } }
I20260620 18:41:24.985744 13262 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2df3a14eb96b4cfa9cb5dd347214d863 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833cb7be5aa0462fbe0003338d84b367" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 35829 } } peers { permanent_uuid: "2df3a14eb96b4cfa9cb5dd347214d863" member_type: VOTER last_known_addr { host: "127.22.37.61" port: 40617 } } peers { permanent_uuid: "31d7504f52674d98964749fdda4e8dde" member_type: VOTER last_known_addr { host: "127.22.37.60" port: 46733 } } }
I20260620 18:41:24.986158 13262 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2df3a14eb96b4cfa9cb5dd347214d863 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:41:24.988034 13139 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2df3a14eb96b4cfa9cb5dd347214d863 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:41:24.988106 13114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "833cb7be5aa0462fbe0003338d84b367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "2df3a14eb96b4cfa9cb5dd347214d863"
I20260620 18:41:24.988595 13114 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2df3a14eb96b4cfa9cb5dd347214d863 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:41:24.993568 13114 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2df3a14eb96b4cfa9cb5dd347214d863 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 833cb7be5aa0462fbe0003338d84b367 in term 1.
I20260620 18:41:24.994285 13113 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2df3a14eb96b4cfa9cb5dd347214d863 [term 1 FOLLOWER]: Refusing update from remote peer 833cb7be5aa0462fbe0003338d84b367: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:41:24.995369 13249 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:41:24.996989 13233 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 833cb7be5aa0462fbe0003338d84b367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2df3a14eb96b4cfa9cb5dd347214d863" member_type: VOTER last_known_addr { host: "127.22.37.61" port: 40617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:41:24.997417 13249 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:41:25.010099 13249 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 833cb7be5aa0462fbe0003338d84b367: Generated new TSK 0
I20260620 18:41:25.011365 13249 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:41:25.031395 13273 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2df3a14eb96b4cfa9cb5dd347214d863 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "833cb7be5aa0462fbe0003338d84b367" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833cb7be5aa0462fbe0003338d84b367" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 35829 } } peers { permanent_uuid: "2df3a14eb96b4cfa9cb5dd347214d863" member_type: VOTER last_known_addr { host: "127.22.37.61" port: 40617 } } peers { permanent_uuid: "31d7504f52674d98964749fdda4e8dde" member_type: VOTER last_known_addr { host: "127.22.37.60" port: 46733 } } }
I20260620 18:41:25.031826 13273 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2df3a14eb96b4cfa9cb5dd347214d863 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:41:25.032205 13262 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2df3a14eb96b4cfa9cb5dd347214d863 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 833cb7be5aa0462fbe0003338d84b367. Latest consensus state: current_term: 1 leader_uuid: "833cb7be5aa0462fbe0003338d84b367" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833cb7be5aa0462fbe0003338d84b367" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 35829 } } peers { permanent_uuid: "2df3a14eb96b4cfa9cb5dd347214d863" member_type: VOTER last_known_addr { host: "127.22.37.61" port: 40617 } } peers { permanent_uuid: "31d7504f52674d98964749fdda4e8dde" member_type: VOTER last_known_addr { host: "127.22.37.60" port: 46733 } } }
I20260620 18:41:25.032471 13262 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2df3a14eb96b4cfa9cb5dd347214d863 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:41:25.034407 13246 sys_catalog.cc:455] T 00000000000000000000000000000000 P 833cb7be5aa0462fbe0003338d84b367 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "833cb7be5aa0462fbe0003338d84b367" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833cb7be5aa0462fbe0003338d84b367" member_type: VOTER last_known_addr { host: "127.22.37.62" port: 35829 } } peers { permanent_uuid: "2df3a14eb96b4cfa9cb5dd347214d863" member_type: VOTER last_known_addr { host: "127.22.37.61" port: 40617 } } peers { permanent_uuid: "31d7504f52674d98964749fdda4e8dde" member_type: VOTER last_known_addr { host: "127.22.37.60" port: 46733 } } }
I20260620 18:41:25.034806 13246 sys_catalog.cc:458] T 00000000000000000000000000000000 P 833cb7be5aa0462fbe0003338d84b367 [sys.catalog]: This master's current role is: LEADER
I20260620 18:41:25.056466 13278 catalog_manager.cc:1259] Loaded cluster ID: ca805dac8b8a49af97960e75931e9183
I20260620 18:41:25.056658 13278 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2df3a14eb96b4cfa9cb5dd347214d863: loading cluster ID for follower catalog manager: success
I20260620 18:41:25.060492 13278 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2df3a14eb96b4cfa9cb5dd347214d863: acquiring CA information for follower catalog manager: success
I20260620 18:41:25.068305 13278 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2df3a14eb96b4cfa9cb5dd347214d863: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 18:41:25.172998 13216 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:41:25.163407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48516 (local address 127.22.37.60:46733)
0620 18:41:25.163569 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:25.163575 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:25.170928 (+ 7353us) server_negotiation.cc:407] Connection header received
0620 18:41:25.171408 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:25.171414 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:25.171522 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:25.171606 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:25.172718 (+ 1112us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48516: BlockingRecv error: recv got EOF from 127.0.0.1:48516 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260620 18:41:25.175105 13144 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:41:25.163225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39686 (local address 127.22.37.61:40617)
0620 18:41:25.163438 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:25.163444 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:25.173580 (+ 10136us) server_negotiation.cc:407] Connection header received
0620 18:41:25.173801 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:25.173807 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:25.173885 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:25.173966 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:25.174870 (+ 904us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39686: BlockingRecv error: recv got EOF from 127.0.0.1:39686 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260620 18:41:25.177223 13069 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:41:25.163055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51936 (local address 127.22.37.62:35829)
0620 18:41:25.163252 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:25.163260 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:25.175657 (+ 12397us) server_negotiation.cc:407] Connection header received
0620 18:41:25.175920 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:25.175927 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:25.176013 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:25.176090 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:25.176991 (+ 901us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51936: BlockingRecv error: recv got EOF from 127.0.0.1:51936 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 18:41:25.227887 13144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:25.212297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39692 (local address 127.22.37.61:40617)
0620 18:41:25.212502 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:25.212509 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:25.219526 (+ 7017us) server_negotiation.cc:407] Connection header received
0620 18:41:25.220084 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:25.220092 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:25.220329 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:25.220496 (+ 167us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:41:25.221279 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:25.222533 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:25.224663 (+ 2130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:25.225303 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:25.225460 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:25.226243 (+ 783us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:41:25.227032 (+ 789us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:41:25.227189 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:25.227455 (+ 266us) server_negotiation.cc:299] Negotiation successful
0620 18:41:25.227548 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260620 18:41:25.274941 13216 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:41:25.261532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48526 (local address 127.22.37.60:46733)
0620 18:41:25.261732 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:25.261739 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:25.272801 (+ 11062us) server_negotiation.cc:407] Connection header received
0620 18:41:25.273349 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:25.273381 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:25.273521 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:25.273604 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:25.274693 (+ 1089us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48526: BlockingRecv error: recv got EOF from 127.0.0.1:48526 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 18:41:25.325457 13144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:25.310405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39706 (local address 127.22.37.61:40617)
0620 18:41:25.310637 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:25.310644 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:25.318218 (+ 7574us) server_negotiation.cc:407] Connection header received
0620 18:41:25.318762 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:25.318768 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:25.318872 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:25.318948 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:41:25.319678 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:25.320853 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:25.323074 (+ 2221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:25.323811 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:25.323913 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:25.324569 (+ 656us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:41:25.324719 (+ 150us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:41:25.324780 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:25.325008 (+ 228us) server_negotiation.cc:299] Negotiation successful
0620 18:41:25.325096 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260620 18:41:25.372269 13216 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:41:25.359338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48534 (local address 127.22.37.60:46733)
0620 18:41:25.359550 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:25.359556 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:25.370184 (+ 10628us) server_negotiation.cc:407] Connection header received
0620 18:41:25.370721 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:25.370727 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:25.370827 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:25.370901 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:25.372009 (+ 1108us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48534: BlockingRecv error: recv got EOF from 127.0.0.1:48534 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 18:41:25.423347 13144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:25.408715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39716 (local address 127.22.37.61:40617)
0620 18:41:25.408919 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:25.408926 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:25.416296 (+ 7370us) server_negotiation.cc:407] Connection header received
0620 18:41:25.416844 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:25.416851 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:25.416948 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:25.417012 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:41:25.417674 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:25.418880 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:25.420976 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:25.421668 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:25.421773 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:25.422502 (+ 729us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:41:25.422651 (+ 149us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:41:25.422710 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:25.422932 (+ 222us) server_negotiation.cc:299] Negotiation successful
0620 18:41:25.423030 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260620 18:41:25.445593 13216 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:41:25.443386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48540 (local address 127.22.37.60:46733)
0620 18:41:25.443616 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:25.443626 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:25.444147 (+ 521us) server_negotiation.cc:407] Connection header received
0620 18:41:25.444364 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:25.444370 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:25.444460 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:25.444571 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:25.445317 (+ 746us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48540: BlockingRecv error: recv got EOF from 127.0.0.1:48540 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 18:41:25.458355 13144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:25.452627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39730 (local address 127.22.37.61:40617)
0620 18:41:25.452931 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:25.452941 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:25.453463 (+ 522us) server_negotiation.cc:407] Connection header received
0620 18:41:25.453691 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:25.453697 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:25.453778 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:25.453890 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:41:25.454330 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:25.455054 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:25.456495 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:25.457038 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:25.457170 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:25.457624 (+ 454us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:41:25.457724 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:41:25.457816 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:25.457967 (+ 151us) server_negotiation.cc:299] Negotiation successful
0620 18:41:25.458042 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150}
W20260620 18:41:25.465759 13216 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:41:25.464136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48546 (local address 127.22.37.60:46733)
0620 18:41:25.464413 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:25.464419 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:25.464438 (+ 19us) server_negotiation.cc:407] Connection header received
0620 18:41:25.464600 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:25.464606 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:25.464689 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:25.464744 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:25.465582 (+ 838us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48546: BlockingRecv error: recv got EOF from 127.0.0.1:48546 (error 108)
Metrics: {"server-negotiator.queue_time_us":144}
I20260620 18:41:25.483017 13144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:25.477168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39738 (local address 127.22.37.61:40617)
0620 18:41:25.477413 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:25.477422 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:25.477975 (+ 553us) server_negotiation.cc:407] Connection header received
0620 18:41:25.478188 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:25.478193 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:25.478271 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:25.478401 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:41:25.478794 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:25.479578 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:25.480893 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:25.481517 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:25.481685 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:25.482147 (+ 462us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:41:25.482271 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:41:25.482372 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:25.482524 (+ 152us) server_negotiation.cc:299] Negotiation successful
0620 18:41:25.482640 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260620 18:41:25.491283 13216 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:41:25.489046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48562 (local address 127.22.37.60:46733)
0620 18:41:25.489301 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:25.489309 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:25.489889 (+ 580us) server_negotiation.cc:407] Connection header received
0620 18:41:25.490099 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:25.490104 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:25.490188 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:25.490296 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:25.491029 (+ 733us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48562: BlockingRecv error: recv got EOF from 127.0.0.1:48562 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 18:41:25.516369 13144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:25.510539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39742 (local address 127.22.37.61:40617)
0620 18:41:25.510806 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:25.510816 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:25.511347 (+ 531us) server_negotiation.cc:407] Connection header received
0620 18:41:25.511573 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:25.511577 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:25.511649 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:25.511756 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:41:25.512166 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:25.513045 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:25.514509 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:25.515016 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:25.515148 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:25.515584 (+ 436us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:41:25.515693 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:41:25.515798 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:25.515955 (+ 157us) server_negotiation.cc:299] Negotiation successful
0620 18:41:25.516043 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
W20260620 18:41:25.524765 13216 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:41:25.522476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48564 (local address 127.22.37.60:46733)
0620 18:41:25.522718 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:25.522726 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:25.523217 (+ 491us) server_negotiation.cc:407] Connection header received
0620 18:41:25.523470 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:25.523474 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:25.523566 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:25.523680 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:25.524510 (+ 830us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48564: BlockingRecv error: recv got EOF from 127.0.0.1:48564 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
I20260620 18:41:25.566651 13144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:25.560587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39748 (local address 127.22.37.61:40617)
0620 18:41:25.560830 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:25.560837 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:25.561221 (+ 384us) server_negotiation.cc:407] Connection header received
0620 18:41:25.561428 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:25.561434 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:25.561541 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:25.561658 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:41:25.562140 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:25.563237 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:25.564521 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:25.565279 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:25.565427 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:25.565816 (+ 389us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:41:25.565961 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:41:25.566054 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:25.566206 (+ 152us) server_negotiation.cc:299] Negotiation successful
0620 18:41:25.566288 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
W20260620 18:41:25.574781 13216 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:41:25.572909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48572 (local address 127.22.37.60:46733)
0620 18:41:25.573109 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:25.573115 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:25.573411 (+ 296us) server_negotiation.cc:407] Connection header received
0620 18:41:25.573563 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:25.573567 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:25.573640 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:25.573722 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:25.574562 (+ 840us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48572: BlockingRecv error: recv got EOF from 127.0.0.1:48572 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
I20260620 18:41:25.647799 13144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:25.642527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39752 (local address 127.22.37.61:40617)
0620 18:41:25.642754 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:25.642762 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:25.643093 (+ 331us) server_negotiation.cc:407] Connection header received
0620 18:41:25.643319 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:25.643323 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:25.643391 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:25.643490 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:41:25.643909 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:25.644659 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:25.646092 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:25.646541 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:25.646693 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:25.647098 (+ 405us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:41:25.647198 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:41:25.647296 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:25.647442 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 18:41:25.647522 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260620 18:41:25.655171 13216 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:41:25.653150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48580 (local address 127.22.37.60:46733)
0620 18:41:25.653403 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:25.653411 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:25.653802 (+ 391us) server_negotiation.cc:407] Connection header received
0620 18:41:25.654010 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:25.654015 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:25.654110 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:25.654218 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:25.654959 (+ 741us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48580: BlockingRecv error: recv got EOF from 127.0.0.1:48580 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 18:41:25.767042 13230 catalog_manager.cc:1259] Loaded cluster ID: ca805dac8b8a49af97960e75931e9183
I20260620 18:41:25.767144 13230 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 31d7504f52674d98964749fdda4e8dde: loading cluster ID for follower catalog manager: success
I20260620 18:41:25.769585 13230 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 31d7504f52674d98964749fdda4e8dde: acquiring CA information for follower catalog manager: success
I20260620 18:41:25.771297 13230 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 31d7504f52674d98964749fdda4e8dde: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 18:41:25.791657 13144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:25.786688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39768 (local address 127.22.37.61:40617)
0620 18:41:25.786947 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:25.786953 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:25.787245 (+ 292us) server_negotiation.cc:407] Connection header received
0620 18:41:25.787417 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:25.787422 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:25.787492 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:25.787573 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:41:25.787977 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:25.788776 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:25.790201 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:25.790591 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:25.790686 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:25.791036 (+ 350us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:41:25.791134 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:41:25.791199 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:25.791305 (+ 106us) server_negotiation.cc:299] Negotiation successful
0620 18:41:25.791364 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 18:41:25.803154 13216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:25.797803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48588 (local address 127.22.37.60:46733)
0620 18:41:25.798066 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:25.798073 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:25.798304 (+ 231us) server_negotiation.cc:407] Connection header received
0620 18:41:25.798453 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:25.798458 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:25.798609 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:25.798703 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:41:25.799020 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:25.799678 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:25.800965 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:25.801410 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:25.801512 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:25.802009 (+ 497us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:41:25.802551 (+ 542us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:41:25.802701 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:25.802740 (+ 39us) server_negotiation.cc:299] Negotiation successful
0620 18:41:25.802807 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260620 18:41:25.826610 13216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:25.809170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48590 (local address 127.22.37.60:46733)
0620 18:41:25.809433 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:25.809439 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:25.817186 (+ 7747us) server_negotiation.cc:407] Connection header received
0620 18:41:25.817707 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:25.817716 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:25.817848 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:25.817950 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:41:25.818975 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:25.820587 (+ 1612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:25.823808 (+ 3221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:25.824567 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:25.824679 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:25.825617 (+ 938us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:41:25.825772 (+ 155us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:41:25.825874 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:25.826163 (+ 289us) server_negotiation.cc:299] Negotiation successful
0620 18:41:25.826258 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 18:41:25.832015 13144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:25.809148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39784 (local address 127.22.37.61:40617)
0620 18:41:25.809444 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:25.809453 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:25.825825 (+ 16372us) server_negotiation.cc:407] Connection header received
0620 18:41:25.826122 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:25.826127 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:25.826217 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:25.826300 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:41:25.827033 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:25.827849 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:25.830015 (+ 2166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:25.830466 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:25.830624 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:25.831120 (+ 496us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:41:25.831282 (+ 162us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:41:25.831382 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:25.831588 (+ 206us) server_negotiation.cc:299] Negotiation successful
0620 18:41:25.831689 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260620 18:41:25.832957 13383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:25.808875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51950 (local address 127.22.37.62:35829)
0620 18:41:25.824290 (+ 15415us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:25.824296 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:25.824965 (+ 669us) server_negotiation.cc:407] Connection header received
0620 18:41:25.825217 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:25.825226 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:25.825537 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:25.825751 (+ 214us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:41:25.826484 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:25.827574 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:25.829864 (+ 2290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:25.830269 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:25.830472 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:25.831047 (+ 575us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:41:25.831579 (+ 532us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:41:25.831908 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:25.831958 (+ 50us) server_negotiation.cc:299] Negotiation successful
0620 18:41:25.832047 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15259,"thread_start_us":16530,"threads_started":1}
I20260620 18:41:25.843549 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 12998
I20260620 18:41:25.844735 13060 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:41:25.959302 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 12998
I20260620 18:41:25.980827 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 13072
I20260620 18:41:25.981997 13134 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:41:26.095669 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 13072
I20260620 18:41:26.116935 22676 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1UXpXs/build/asan/bin/kudu pid 13146
I20260620 18:41:26.118083 13208 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:41:26.227598 22676 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1UXpXs/build/asan/bin/kudu with pid 13146
2026-06-20T18:41:26Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4354 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (8790 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (358763 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST